Closed AddictArts closed 10 months ago
In 1.5 we have toggled on using the daemon by default. On your machine the connection between the main turbo process and the daemon is failing.
This does not prevent your build from running to completion but we do need to identify why that connection is failing.
@nathanhammond Thanks for the reply. How do you even start the daemon? I searched the docs and only the cache config mentions a daemon. I understand the principle, but I just upgraded, didn't make a change to turbo.json to run a daemon. Is this something like Gradle where it spins it up on first launch? Just wanted to mention that some confusion for me is the lack of docs.
I also found the option --no-daemon
to turn the warning off. The issue though is that the process identified in turbo.pid doesn't exist. Not running. So, maybe it tries to launch a daemon and can't?
* Can you tell me more about your OS?
Edition Windows 10 Pro for Workstations Version 21H2 Installed on 11/24/2021 OS build 19044.2006 Experience Windows Feature Experience Pack 120.2212.4180.0
* Can you identify if you have write permissions to the path generated here? https://github.com/vercel/turborepo/blob/0dfa8e681c64a2bba9e8227d349b87f2f4bf7d04/cli/internal/daemon/daemon.go#L47 (we need better logging here to make this easier, you'd need to identify where Golang tempdir is creating a folder)
%HOME%\AppData\Local\Temp
full permission. Also turbo creates a turbod folder there and turbod.pid
and turbod.sock
.
* what permissions does your user have?
For me that hexHash join is Temp\turbod\5ad724522893520a
I am fairly certain I have full permission. I run other go executables and go tools without issue. I can also open and edit the pid file. The sock file is 0 bytes and the pid file has a pid, but when I run that pid is not running, exluding --no-daemon
. This makes the warning inaccurate. The log shows no error.
It should be WARNING: Failed to start turbod
, because that seems to be what is happening. Something needs to go into the log to say why, which is also not happening.
the same warning here on Windows 11
turbo daemon status
should give a location for logs from the daemon. If you could post those, it would be helpful to figure out why the daemon is not running for is unable to be contacted.
Hi there, also getting the same issue. On trying turbo daemon status
I get this:
❯ pnpm exec turbo daemon status
Failed to contact daemon: connection to turbo daemon process failed. Please ensure the following:
- the process identified by the pid in the file at /tmp/turbod/0f771c1e12c2b3b3/turbod.pid is not running, and remove /tmp/turbod/0f771c1e12c2b3b3/turbod.pid
- check the logs at /home/jayg-hive/.local/share/turborepo/logs/0f771c1e12c2b3b3-Prosper-Ultimate.log
- the unix domain socket at /tmp/turbod/0f771c1e12c2b3b3/turbod.sock has been removed
You can also run without the daemon process by passing --no-daemon
I'm on WSL2 with the following details:
WSL version: 0.67.6.0
Kernel version: 5.15.62.1
WSLg version: 1.0.44
MSRDC version: 1.2.3401
Direct3D version: 1.606.4
DXCore version: 10.0.25131.1002-220531-1700.rs-onecore-base2-hyp
Windows version: 10.0.22621.601
Nothing out of the ordinary on the logs, they just say something about fsnotify
events on my monorepo files:
2022-09-23T09:27:16.339+0800 [DEBUG] turbod.rpc server.GlobWatcher: Got fsnotify event {/home/jayg-hive/hive-gaming/**redacted**/packages/utils/.turbo/turbo-test.log 3}
Same issue for me on Mac, today installed turbo for first time
WARNING failed to contact turbod. Continuing in standalone mode: connection to turbo daemon process failed. Please ensure the following:
- the process identified by the pid in the file at /var/folders/b2/7xflcg550111jtksj2sb32n00000gn/T/turbod/5c8a39790d3101dc/turbod.pid is not running, and remove /var/folders/b2/7xflcg550111jtksj2sb32n00000gn/T/turbod/5c8a39790d3101dc/turbod.pid
- check the logs at /Users/marius/Library/Application Support/turborepo/logs/5c8a39790d3101dc-test__turborepo.log
- the unix domain socket at /var/folders/b2/7xflcg550111jtksj2sb32n00000gn/T/turbod/5c8a39790d3101dc/turbod.sock has been removed
You can also run without the daemon process by passing --no-daemon
Error from log is
2022-09-23T10:48:48.558+0700 [ERROR] turbod: error: EXTRA_VALUE_AT_END="watching /Volumes/Media/Dev Projects/Projects/test__turborepo: timed out waiting for initial fsevents cookie: filewatching failed to start"
Gets the same warning message as @MariuzMM and @jayg-hive
Been using turbo for a while now. This warning started in version 1.5.2
On Windows10:
2022-09-23T14:16:10.471+0200 [ERROR] turbod: error: EXTRA_VALUE_AT_END="failed to lock the pid file at . Is another turbo daemon running?: Locked by other process"
No error on WSL2/Ubuntu :)
I'm seeing the same after a sweeping upgrade of dependencies today. Log output is a repeat of:
2022-09-23T09:42:53.645-0400 [ERROR] turbod: error: EXTRA_VALUE_AT_END="watching /Volumes/Work/code/MYPROJECT: timed out waiting for initial fsevents cookie: filewatching failed to start"
Console is outputting:
yarn turbo daemon status
yarn run v1.22.19
$ /Volumes/Work/code/MYPROJECT/node_modules/.bin/turbo daemon status
Failed to contact daemon: the daemon is not running
✨ Done in 0.32s.
Reverting and hard-coding to "turbo": "1.5.1"
in package.json
"solves" this issue.
Couple things going on:
EXTRA_VALUE_AT_END
. I'll fix it, but that part is safe to ignoreturbo
.For those of you hitting the warning, can you confirm that turbo
is still producing the correct results, just with the extra warning? Or is this actively breaking anyone? If so, I'd like to prioritize addressing breakage.
@gkn if you run turbo daemon status
, it should give you the path to the .pid
file? Can you check if there is in fact a process with that PID? Anything in the daemon logs (also from turbo daemon status
) ?
Running MacOS 12.6 here. When running the command turbo daemon status
, I get "Failed to contact daemon: the daemon is not running." It's not preventing building for me, just giving the warning.
Addressing EXTRA_VALUE_AT_END
: #2068
Additional context and debugging thoughts:
turbo daemon status
is not intended to start the daemon. If it's not running, that's the status and that's fine. The daemon also does not stay alive forever, it has an idle timeout. The turbo
cli will attempt to restart it next time it needs it.turbo daemon
in your monorepo root will run the daemon in the foreground and log to the terminal.turbo daemon --help
Also, to everyone, thanks for your patience while we iron out the edge cases on this. There may be platforms and scenarios where the daemon does not work or does not provide benefit, and we'll work to detect those and behave accordingly as we identify them.
Hopefully this will help. I run the daemon and no errors to the console etc. Looks good using turbo daemon
.
When I run turbo run dev
the turbod dies. Last log message is
2022-09-23T14:04:32.896-0700 [DEBUG] turbod.rpc server.GlobWatcher: Got fsnotify event {oranges\apps\web\.turbo\turbo-lint.log451477266 4}
Couple things going on:
- Minor, display-only bug related to
EXTRA_VALUE_AT_END
. I'll fix it, but that part is safe to ignore- re: cookie timeouts, I'm going to bump it a little, but if it takes too long, it's likely better to for us skip filewatching.
- on that topic, the warning should be downgraded to informational. Filewatching is an optional optimization, and is not necessary for the proper functioning of
turbo
.For those of you hitting the warning, can you confirm that
turbo
is still producing the correct results, just with the extra warning? Or is this actively breaking anyone? If so, I'd like to prioritize addressing breakage.
Yup, no breakage whatsoever. It still runs the intended task. 👍
On Windows 10: 1) I cleared the logs and pid + cookie file and folders in the user folder.
I still get the same warning message but now get a log file without error messages, just a lot of fsnotify messages. It even includes folders that is not specified as part of npm workspaces. This seems odd and wasteful, especially since it will watch node_modules in that folder too:
2022-09-24T08:41:40.051+0200 [DEBUG] turbod.rpc server.fsnotify: watching directory C:\xxxx\tools\storybook\node_modules\@mrmlnc
Initial startup seems slow.
Monorepo is relatively large, a few hundred folders with roughly 5000 source files.
Tried manual start of turbod: $ npx turbo daemon start connection to turbo daemon process failed. Please ensure the following: ... ...
Size of log file
I primarily use WSL2/Ubuntu for this project. I noticed that the size of the turbod log file was large. (roughly 1.5 million lines, 260 MB) in just a few turbo runs since yesterday. Seems odd.
Turning down daemon logging: #2085
@gkn Thanks for the details. Better scoping filewatching sounds like an area for improvement.
Everything still seems to work fine, but it throws warnings on every run.
The logging and EXTRA_VALUE_AT_END
fixes are now available in turbo@1.5.4-canary.0
Same here, I'm running turbo@1.5.4
on arm macos 12.5.1.
WARNING failed to contact turbod. Continuing in standalone mode: connection to turbo daemon process failed. Please ensure the following:
- the process identified by the pid in the file at /var/folders/ls/sq7g081s1g3cvtlz41qp8z2m0000gn/T/turbod/fe25f7293c19b4d8/turbod.pid is not running, and remove /var/folders/ls/sq7g081s1g3cvtlz41qp8z2m0000gn/T/turbod/fe25f7293c19b4d8/turbod.pid
- check the logs at /Users/Andrzej_Klapec/Library/Application Support/turborepo/logs/fe25f7293c19b4d8-tui-frontend-monorepo.log
- the unix domain socket at /var/folders/ls/sq7g081s1g3cvtlz41qp8z2m0000gn/T/turbod/fe25f7293c19b4d8/turbod.sock has been removed
You can also run without the daemon process by passing --no-daemon
I ran yarn upgrade-interactive --latest
to update turbo (previously I had it at 1.4.3) and ran the codemod npx @turbo/codemod migrate-env-var-dependencies
which updated my turbo.json
file and created a new .turbo-cookie
file. Is the .turbo-cookie
file necessary and should it be committed into the repository?
Gets the same warning, and the log file is empty.
On Windows 11, turbo 1.5.4
@klapec the .turbo-cookie
is not necessary, it is intended to be in the directory that turbo uses for config data, not your repository. Did it appear at the root of your repo? And was it created by the codemod?
@gsoltis Thanks for the response. It did appear at the root of my monorepo, I assume it was either created by upgrading turbo to 1.5.4 (postinstall?) or by the codemod. I haven't seen exactly when was it created, I noticed it when I was about to commit my changes to the repository.
@klapec got it. we're looking into how it could've ended up in that spot, but it's safe to delete. Sorry about that!
For the windows users in this thread: it looks like we have an issue somewhere between Go, windows support for AF_UNIX
, and grpc. I'm tracking down the exact fix, but I expect that's the cause of being unable to use the daemon thus far
We've published turbo@1.5.5
which disables the daemon on windows while we sort out the AF_UNIX
socket issue. I expect it will be re-enabled soon, as we've identified the issue and are working on a mitigation.
For those curious about the nitty-gritty details:
url.Parse
to determine how to connect to a serverPath
portion of a url start with a leading /
Path
portion of a url for the location to Dial when the network is unix
/ url scheme is unix://
/
This means that we are either sending in a url that fails to parse (no leading '/'), or a url that parses, but has an extra leading '/'.
Not to worry though, we can patch it to do special handling on windows. Coming soon.
@gsoltis Tried out turbo@1.5.5 on both WSL2/Ubuntu and on the (disabled) Windows variant now. No noticeable issues - good work :)
If you have the time, could you please explain where we actually benefit from utilizing the daemon? I cannot say I notice any differences and could not find any documentation addressing that btw.
@gsoltis : after upgrading Turborepo to version 1.5.5 I still notice the same warning message on my Macbook Pro M1 (MacOS 12.5.1).
A .turbo-cookie
file is created at the root of my project everytime I run a turbo command.
As already mentioned, the warning message is :
WARNING failed to contact turbod. Continuing in standalone mode: connection to turbo daemon process failed. Please ensure the following:
- the process identified by the pid in the file at /var/folders/33/3dflr1_x01b07gpkyb7qsxtc0000gp/T/turbod/ba68216b5b606bf5/turbod.pid is not running, and remove /var/folders/33/3dflr1_x01b07gpkyb7qsxtc0000gp/T/turbod/ba68216b5b606bf5/turbod.pid
- check the logs at /Users/{user}/Library/Application Support/turborepo/logs/ba68216b5b606bf5-{project}.log
- the unix domain socket at /var/folders/33/3dflr1_x01b07gpkyb7qsxtc0000gp/T/turbod/ba68216b5b606bf5/turbod.sock has been removed
You can also run without the daemon process by passing --no-daemon
Note that I have no trouble whatsoever with my pipeline. Everything seems to work accordingly.
@gkn Absolutely, and I apologize it hasn't made it into the docs yet.
The daemon is purely a performance optimization, and not at all necessary for correct functioning of turbo
. The daemon gives turbo
the ability to do some work ahead of time or after a turbo run
without blocking the user. The first feature that it implements today is skipping cache restoration if the outputs haven't changed since the last time a task was run. The result is faster cache hits, since they don't incur any file I/O. Some of the next features that are coming include:
turbo run
doesn't need to wait for uploading turbo run
doesn't need to wait to calculate what has changed, again avoiding file I/O during the time that a user is waitingFurthermore, the daemon has an idle timeout and is disabled automatically in environments that don't "look" like dev machines: terminal is not a tty, presence of some known CI environment variables, etc.
@fveauvy can you give an example of how you are invoking turbo
when you see the .turbo-cookie
file show up? We are looking into it, but so far have not managed to reproduce the issue.
cc @mehulkar
@fveauvy @klapec I've filed https://github.com/vercel/turborepo/issues/2134 for the .turbo-cookie
issue. I'm not sure it's related to the daemon outputs. Please feel free to add any context/additional info about your cases there!
windows users following along, re-enabling the daemon and fixing addressing of the AF_UNIX
socket: #2137
Mac users, @fveauvy @MariuzMM @klapec @richardtorres314 , if you don't mind sharing, how are you invoking turbo
? Is it via npx turbo ...
? Is it in a script in package.json
? If so, which package manager are you using? Some other means?
Any errors in the logs (location reported in the error message)?
I know we have one report of a timeout waiting for a filesystem cookie, anything else?
A small plug here: maybe you can consider using https://github.com/unional/fixture#command-testing or something similar. It allows you to write command line based tests. 🍻
@fveauvy can you give an example of how you are invoking
turbo
when you see the.turbo-cookie
file show up? We are looking into it, but so far have not managed to reproduce the issue.cc @mehulkar
I was planning to record a gif showing you the issue, and I noticed something important regarding this matter.
In my package.json
, the command yarn build
will run turbo run build
. FYI, I use yarn@1.22.19.
When I ran yarn build
in the integrated terminal in VSCode there wasn't any warning message, and the file .turbo-cookie
wasn't created.
But if I run yarn build
in a dedicated terminal (I tried with hyper.js or in the default Terminal of macOS) I can see the warning message and the .turbo-cookie
is created in the root of my project.
@gsoltis : That's all I have in my logs.
2022-10-01T11:19:33.942+0200 [ERROR] turbod: error watching /Users/fveauvy/Code/Repo/pragma-api: timed out waiting for initial fsevents cookie: filewatching failed to start
2022-10-01T11:19:34.941+0200 [ERROR] turbod: error watching /Users/fveauvy/Code/Repo/pragma-api: timed out waiting for initial fsevents cookie: filewatching failed to start
2022-10-01T11:19:35.943+0200 [ERROR] turbod: error watching /Users/fveauvy/Code/Repo/pragma-api: timed out waiting for initial fsevents cookie: filewatching failed to start
@gsoltis I have turbo 1.5.3 installed globally. I run my dev
script from the root of my project which runs turbo run dev --parallel
.
I'm using npm v8.1.0.
I only receive a warning message
failed to contact turbod. Continuing in standalone mode: connection to turbo daemon process failed. Please ensure the following:
- the process identified by the pid in the file at /var/folders/1t/ps8q7_ys0gdf21b2hqdlzmf00000gn/T/turbod/b7bf2d52b7e43eff/turbod.pid is not running, and remove /var/folders/1t/ps8q7_ys0gdf21b2hqdlzmf00000gn/T/turbod/b7bf2d52b7e43eff/turbod.pid
- check the logs at /Users/richardtorres/Library/Application Support/turborepo/logs/b7bf2d52b7e43eff-class-recipes.log
- the unix domain socket at /var/folders/1t/ps8q7_ys0gdf21b2hqdlzmf00000gn/T/turbod/b7bf2d52b7e43eff/turbod.sock has been removed
You can also run without the daemon process by passing --no-daemon
Everything else works fine for me as of right now.
i'm having similar issues and it's also creating a .turbo-cookie file as the root. Problem I have, it's actually stopping my app from fully running, not allowing my localhost to connect. I am running
"turbo run dev --no-cache --parallel --continue --filter=!solace --filter=!cms --filter=!graphql-internal
"turbo": "1.5.3",
"npm": ">=7.0.0",
"node": ">=14.0.0"
getting an issue of
$ turbo run build --include-dependencies --filter=!mastermind --filter=!cms --filter=!graphql-internal --filter=!solace --filter=!esmodule-app --filter=!docs
WARNING failed to contact turbod. Continuing in standalone mode: connection to turbo daemon process failed. Please ensure the following:
- the process identified by the pid in the file at /var/folders/yv/yrw7qjks30l6p3vfyfr7c4hm0000gn/T/turbod/cb043b36610831c5/turbod.pid is not running, and remove /var/folders/yv/yrw7qjks30l6p3vfyfr7c4hm0000gn/T/turbod/cb043b36610831c5/turbod.pid
- check the logs at /Users/elvisplaza/Library/Application Support/turborepo/logs/cb043b36610831c5-monorepo.log
- the unix domain socket at /var/folders/yv/yrw7qjks30l6p3vfyfr7c4hm0000gn/T/turbod/cb043b36610831c5/turbod.sock has been removed
You can also run without the daemon process by passing --no-daemon
@elvisplaza as a workaround, can you try passing --no-daemon
. I'm also curious how you're invoking turbo
. Do you have it globally installed or is it a dependency in your repository? Is it being invoked via a script from package.json
?
Re: connection failure, I suspect that's something else happening, since the daemon does not bind localhost
. Can you check your output to verify that your server is running and that there is not also some other error?
In my repo there is a dir that is git ignored and auto managed by a docker container. Turbo repo's daemon is trying to watch this directory and dieing since it is trying to watch it. Removing the specific files did seem to fix the issue with it not starting up so must be related.
Can the turbo repo not try to watch any files that are git ignored? Or allow somewhere to specify that I don't want it watching specific files? At very least not die if permission issues occur?
1.5.5 disables daemon on windows, but it still checks for daemon during run. e.g.:
Failed to check if we can skip restoring outputs for @just-web/preferences#coverage: rpc error: code = Unknown desc = timed out waiting for cookie. Proceeding to check cache
Failed to mark outputs as cached for @just-web/preferences#coverage: rpc error: code = Unknown desc = timed out waiting for cookie
cache hit, replaying output
They waste quite a bit of time:
Tasks: 34 successful, 34 total
Cached: 34 cached, 34 total
Time: 9.878s >>> FULL TURBO
with --no-daemon
:
Tasks: 34 successful, 34 total
Cached: 34 cached, 34 total
Time: 678ms >>> FULL TURBO
@unional That looks like the daemon was not actually disabled. That error message in particular comes from the daemon code.
Can you share your platform information? Where this happened, and verify the turbo --version
?
Also note that 1.5.6
adds the daemon back for windows and should properly connect over a unix domain socket.
@unional That looks like the daemon was not actually disabled. That error message in particular comes from the daemon code.
Can you share your platform information? Where this happened, and verify the
turbo --version
?Also note that
1.5.6
adds the daemon back for windows and should properly connect over a unix domain socket.
It fails still on Windows 10. No daemon and constant pestering WARNING message. Back to --no-daemon
.
@AddictArts sorry to hear that, do you by any chance know the build # for the version of windows you're running?
My understanding is that anything after 17063 should work, but if you have an older version we can look at better ways to detect support.
The error is from 1.5.6, on Windows 11
verified version 1.5.6 via pnpm turbo --version
encountering same daemon warning
win 10 OSbuild: 19044.2006
@gsoltis Above, you have other reports for Windows 11 and 10 of the daemon not working in 1.5.6 which unfortunately turned it back on.
Edition Windows 10 Pro for Workstations Version 21H2 Installed on 11/24/2021 OS build 19044.2130 Experience Windows Feature Experience Pack 120.2212.4180.0
@ y'all: thank you very much for the reports and your patience as we're trying to sort through this.
The biggest problems we have with this will always be our ability to reproduce the issues. Once we know how to reproduce issues we are able to quickly resolve them.
To that end, we always have to review each of the following possible contributing factors:
turbo
version.If you're finding this issue please include as much information as possible into your report. If you're seeing this issue for the first time, please consider opening a new issue instead of tagging onto this one so that we're better able to focus the conversations on the particular issues you report. At this point we suspect that they're very unlikely to be the same contributing factors as other people.
The number of places we have to look for issues is the combinatoric set of all of those inputs. For some large portion of issues we will never be able to reproduce the exact environment and have to rely on detailed reports of behavior from users, and for those we're grateful. We wouldn't have shipped it if we didn't believe that we were close.
Our conditions for turning off the daemon in the future will be based upon, "is this preventing a correct build execution?" We've taken great pains to ensure that it does not interfere with correctness and because of that we do not expect to ever make it opt-in again.
With regards to the warning, it's an important part of:
Since it doesn't interfere with correct operation of turbo
we also won't be removing it. For those of you for whom the daemon regularly fails and the warning bothers you, you have the option of using turbo
with --no-daemon
which will, as a side effect, silence the warning.
A couple of bonus debugging steps. For anybody who gets this warning:
turbo daemon status
.log
file, the .pid
file and the .sock
file exist. Include this information in your report.pid
file is running. Include this information in your report..pid
file..log
file, the .pid
file, and the .sock
file.Then:
turbo daemon start
. This reduces the number of variables we need to consider inside of the Turborepo codebase.turbo daemon status
. Include the output of this command in your report..log
file, the .pid
file and the .sock
file exist. Include this information in your report..pid
file is running. Include this information in your report..log
file and include this information in your report.If this doesn't produce an error message (possible!) include that information in your report.
Finally:
npx create-turbo@latest
.my-turborepo
.This will hopefully help us identify if the issue is related to the contents of your repository or something else.
If you're willing, I will pair with anybody who is able to consistently reproduce a daemon issue and is willing to set up a Turborepo development environment on their machine. You can reach DM me (nathanhammond) on our Discord: https://turborepo.org/discord. This is the surest way to get a resolution.
I'm in UTC+8, so we'll probably go back and forth a bit before finding a time to pair. It'll likely also be multiple sessions unless you set up a development environment prior to our call. (This is not a prerequisite! I'll be happy to work with you on that process.)
Note that for those of you on Windows I will be less familiar with your environment, so it'll be a little bit of banging rocks together until it works, but we'll get there.
@kalvenschraut Can you tell me about the permissions on appendonlydir
? And what OS you're on?
@\everybody else: I do not have enough information for me to begin to identify issues. Please collect information from my previous post and drop it in this issue or reach out on Discord so that I can help diagnose!
What version of Turborepo are you using?
1.5.1
What package manager are you using / does the bug impact?
npm
What operating system are you using?
Windows
Describe the Bug
After upgrading from 1.4.7 I now get a warning. Following the warning text leads to not a fix. The log is fine and I've removed the temp files, no fix.
Expected Behavior
No warning or turbod properly launching. If a warning must be shown, it should reflect failure to launch turbod, not contact it.
To Reproduce
Upgrade a 1.4.7 to 1.5.1