Closed OvermindDL1 closed 8 years ago
Would you mind helping us out debug this since you are on a windows machine? Can you add a sleep at the File.rm_rf! that fails and check with process explorer if mix has an open handle to the directory?
You can install the development version of hex by calling mix install
from the project directory. Run mix local.hex
to revert to stable. Thank you!
Following this error as it happens many many times a day (not able to replicate the other issue at the moment it seems):
** (File.Error) could not remove files and directories recursively from "c:/Users/<myuser>/Projects/my_server/_build/dev/lib/my_server/priv": file already exists
(elixir) lib/file.ex:919: File.rm_rf!/1
(mix) lib/mix/utils.ex:317: Mix.Utils.symlink_or_copy/2
(mix) lib/mix/project.ex:414: Mix.Project.build_structure/2
(phoenix) lib/phoenix/code_reloader/server.ex:111: Phoenix.CodeReloader.Server.mix_compile/1
(phoenix) lib/phoenix/code_reloader/server.ex:83: Phoenix.CodeReloader.Server.mix_compile/3
(phoenix) lib/phoenix/code_reloader/server.ex:40: anonymous fn/2 in Phoenix.CodeReloader.Server.handle_call/3
(phoenix) lib/phoenix/code_reloader/server.ex:132: Phoenix.CodeReloader.Server.proxy_io/1
(phoenix) lib/phoenix/code_reloader/server.ex:38: Phoenix.CodeReloader.Server.handle_call/3
(stdlib) gen_server.erl:615: :gen_server.try_handle_call/4
(stdlib) gen_server.erl:647: :gen_server.handle_msg/5
(stdlib) proc_lib.erl:247: :proc_lib.init_p_do_apply/3
That would be in file lib/mix/utils.ex
, however there is no such file in hex, and overwriting a stock Elixir file sounds painful. Hmm, I could :int
terrupt it but it gets called a lot* and the issue only happens sporadically...
Hmm, I might be able to hot-load a replacement copy in, let me try that...
That error seems to be from phoenix code reloader. It's not the same as the one you pasted in the issue.
I posted both in the other thread and both in the top post, it was the same issue in two different areas, one where hex was updating deps (which it is doing fine now), and the other through phoenix's reloader (randomly through-out the day, many times).
The error you get from mix deps.get
is from hex. The error you get when running your application is from mix. It might be easier to debug the latter error instead since it wont require you to install the development version of hex.
You would have to either install a development version of hex or a development version of elixir.
Hrmm, I managed to cause it again (and I had an injected replacement File.ex to catch it right when it is called). Without the timeout it is throwing the error every time (I managed to get the program in a state again where it needs to update the files in priv), with the timeout it is never throwing the error and it is successful. Attempting to search for a process that is accessing that directory as fast as I can never reveals any process touching that directory. So... it really seems like a race condition... I bet it is the same with hex if I can manage to duplicate its issue again...
I managed to cause mix deps.compile
to cause a similar error now (via File.cp_r
! I was trying to replicate the hex error by downgrading and upgrading packages (phoenix specifically) and managed to cause this after a mix deps.get
then mix deps.compile
:
$ mix deps.compile
===> Compiling idna
** (File.CopyError) could not copy recursively from "c:/Users/<user>/Projects/my_server/deps/idna/ebin" to "c:/Users/<user>/Projects/my_server/_build/dev/lib/idna/ebin". c:/Users/<user>/Projects/my_server/_build/dev/lib/idna/ebin: permission denied
(elixir) lib/file.ex:592: File.cp_r!/3
(mix) lib/mix/utils.ex:328: Mix.Utils.do_symlink_or_copy/3
(mix) lib/mix/project.ex:405: Mix.Project.build_structure/2
(elixir) lib/file.ex:1145: File.cd!/2
(mix) lib/mix/tasks/deps.compile.ex:221: anonymous fn/3 in Mix.Tasks.Deps.Compile.build_structure/2
(elixir) lib/enum.ex:651: Enum."-each/2-lists^foreach/1-0-"/2
(elixir) lib/enum.ex:651: Enum.each/2
(mix) lib/mix/tasks/deps.compile.ex:87: anonymous fn/4 in Mix.Tasks.Deps.Compile.compile/2
Copying the directory manually via the same terminal works with no problem, and process explorer shows no holds on either of those directories by any program.
EDIT: And do note, it worked fine the next time I ran mix deps.compile
.
And got it again with another mix deps.compile
(I do not normally run deps.compile manually, rather just compile everything and never touch it again, this took a few more downgrades/upgrades again):
$ mix deps.compile
==> logger_file_backend
Compiling 1 file (.ex)
warning: variable list is unused
lib/logger_file_backend.ex:104
Generated logger_file_backend app
==> base64url (compile)
===> Compiling mimerl
==> fs (compile)
** (File.CopyError) could not copy recursively from "c:/Users/<user>/Projects/my_server/deps/fs/ebin" to "c:/Users/<user>/Projects/my_server/_build/dev/lib/fs/ebin". c:/Users/<user>/Projects/my_server/_build/dev/lib/fs/ebin: permission denied
(elixir) lib/file.ex:592: File.cp_r!/3
(mix) lib/mix/utils.ex:328: Mix.Utils.do_symlink_or_copy/3
(mix) lib/mix/project.ex:405: Mix.Project.build_structure/2
(elixir) lib/file.ex:1145: File.cd!/2
(mix) lib/mix/tasks/deps.compile.ex:221: anonymous fn/3 in Mix.Tasks.Deps.Compile.build_structure/2
(elixir) lib/enum.ex:651: Enum."-each/2-lists^foreach/1-0-"/2
(elixir) lib/enum.ex:651: Enum.each/2
(mix) lib/mix/tasks/deps.compile.ex:87: anonymous fn/4 in Mix.Tasks.Deps.Compile.compile/2
EDIT: And yes, again another mix deps.compile
worked without issue.
@OvermindDL1 You could try Process Monitor to try to identify the "other" process accessing the file. See http://hanselman.com/blog/howtouseprocessmonitorandprocessexplorer.aspx
@OvermindDL1 You could try Process Monitor to try to identify the "other" process accessing the file. See http://hanselman.com/blog/howtouseprocessmonitorandprocessexplorer.aspx
Have checked, it's only ever BEAM, even the anti-virus is disabled for that directory tree.
Anything more we can do on the hex side or can I close this?
I've no clue, I think it is just windows file system being moronic. Maybe adding a loop to 'try again' if you get that error with a 500ms sleep or so for a few times might help?
@OvermindDL1 Please send a PR with that change if you'd like since you are the best person to test it.
@OvermindDL1 Please send a PR with that change if you'd like since you are the best person to test it.
Someday it may bug me enough that I will do that. ^.^
/me really hopes we get linux desktops at work before then...
Hope this helps:
Related Error Messages from Elixir mix on a Windows 10 machine: (File.Error) could not remove files and directories recursively not owner (File.CopyError) could not copy recursively from Could not freeze ./css/app.scss: Cannot read property 'hash' of undefined
Apparent Cause: the project directory was moved. Solution Step 1: Installed WSL Windows Subsystem for Linux and Kali. Solution Step 2: In Kali, performed "mix deps.get",, "mix deps.compile" and "mix phx.server". Solution Step 3: In Windows, performed "mix deps.get", "mix.deps.compile", "assets/npm install" and "mix phx.server". Observation: Apparently mix should not be called from Kali once you begin Step 3.
Same error on macOS Catalina (10.15.6) - however, when I run mix deps.get
again it works w/o errors.
I deleted the _build
and deps
folder to get a clean rebuild of the project, and this happened:
* Getting phoenix (Hex package)
** (File.Error) could not remove files and directories recursively from "~/my_project/deps/phoenix": file already exists
(elixir 1.10.2) lib/file.ex:1288: File.rm_rf!/1
(hex 0.20.5) lib/hex/scm.ex:158: Hex.SCM.update/1
(hex 0.20.5) lib/hex/scm.ex:240: Hex.SCM.checkout/1
(mix 1.10.2) lib/mix/dep/fetcher.ex:64: Mix.Dep.Fetcher.do_fetch/3
(mix 1.10.2) lib/mix/dep/converger.ex:190: Mix.Dep.Converger.all/9
(mix 1.10.2) lib/mix/dep/converger.ex:123: Mix.Dep.Converger.all/7
(mix 1.10.2) lib/mix/dep/converger.ex:108: Mix.Dep.Converger.all/4
(mix 1.10.2) lib/mix/dep/converger.ex:51: Mix.Dep.Converger.converge/4
These are the dependencies:
[
{:phoenix, "1.5.4"},
{:phoenix_pubsub, "~> 2.0"},
{:phoenix_ecto, "~> 4.1"},
{:ecto_sql, "~> 3.4"},
{:postgrex, "~> 0.15.0"},
{:phoenix_html, "~> 2.14.0"},
{:phoenix_live_reload, "~> 1.2", only: :dev},
{:gettext, "~> 0.11"},
{:jason, "~> 1.0"},
{:plug_cowboy, "~> 2.1"},
{:comeonin, "~> 5.1"},
{:argon2_elixir, "~> 2.0"},
{:secure_random, "~> 0.5"},
{:market_prices, "~> 0.2.0", git: "our private repo", tag: "0.2.0"},
{:rocket_carrier, git: "our private repo", tag: "0.9.4"},
{:phoenix_live_view, "~> 0.14.4"},
{:phoenix_live_dashboard, "~> 0.2.6"},
{:telemetry_poller, "~> 0.4"},
{:telemetry_metrics, "~> 0.4"},
{:mox, "~> 0.5.2", only: :test},
{:ex_machina, "~> 2.3", only: :test},
{:assertions, "~> 0.18.1"},
{:floki, "~> 0.27", only: :test},
{:excoveralls, "~> 0.10", only: :test},
{:ecto_enum, "~> 1.4"},
{:neuron, "~> 2.0"},
{:eqrcode, "~> 0.1.6"},
{:pdf_generator, ">=0.4.0"},
{:httpoison, "~> 1.6.2"},
{:exq, "~> 0.14.0"},
{:exq_ui, "~> 0.11", except: :test},
{:ex_aws, "~> 2.1"},
{:ex_aws_s3, "~> 2.0.2"},
{:sweet_xml, "~> 0.6"},
{:observer_cli, "~> 1.5"},
{:swoosh, "~> 1.0.0"},
{:html_entities, "~> 0.4"},
{:nice_maps, "~> 0.3.0"},
{:number, "~> 1.0.1"},
{:hackney, "1.15.2", override: true},
{:edeliver, ">= 1.8.0"},
{:distillery, "~> 2.1", warn_missing: false},
{:countries, "~> 1.6"},
{:ex_url, "~> 1.2"}
]
hex info: Hex v0.20.5
phoenix (the package that failed): 1.5.4
It's not a big deal as long as it still installs all the deps, but would be nice to know why it keeps happening.
It's not a big deal as long as it still installs all the deps, but would be nice to know why it keeps happening.
How do you resolve the issue? The error should prevent the package from being extracted into your deps/ directory.
We need more information to investigate this issue since it only happens for a small set of users and we currently cannot reproduce it. Are you using any non-common filesystem or disk device like a network hard drive or USB stick? What's the permissions for the deps/ directory and its subdirectories? Does it happen for all dependencies or only phoenix? Do you remember any changes you made around the time the error started happening such as upgrading any macOS/Erlang/Elixir versions?
@ericmj thanks for getting back to me about this. I have to try a few things and check what exactly is fixing it, because right now it looks like it just fixes itself somehow. I'll update this comment with more details as soon as I know exactly how to reproduce the issue.
EDIT:
Ok, so now I can't reproduce it anymore. Did the exact same steps as yesterday: delete deps folder, delete _build folder, run mix deps.get
. I am even using the same zsh session as yesterday (computer was on sleep, no restart inbetween.)
However, I am using ElixirLS on atom, which usually automatically fetches the dependencies in the background https://share.getcloudapp.com/wbu7XRol - sometimes it kind of breaks doing that though and I have to run mix deps.get
manually, which is what happened yesterday. Could there have been a conflict with 2 separate entities trying to fetch dendencies?
The other thing that changed since yesterday: I ran mix deps.clean phoenix
and mix deps.compile phoenix
on another project using the same phoenix version, because it had the same error described in this issue. I don't think that is what fixed it, but I figured I should mention it.
To answer your questions:
Are you using any non-common filesystem or disk device like a network hard drive or USB stick?
No, just the regular built in SSD on my Mac
What's the permissions for the deps/ directory and its subdirectories?
https://share.getcloudapp.com/nOukXAeR
Does it happen for all dependencies or only phoenix?
Only happened for phoenix, on 2 separate projects.
Do you remember any changes you made around the time the error started
I ran into the error for the first time this week, no changes I can think of. There was an OS update in between the first occurrence and yesterday (I only ran into this three times, twice yesterday and once monday or tuesday this week.)
Additional infos:
Elixir is installed with asdf, .tool-versions
file
erlang 22.2
elixir 1.10.2-otp-22
nodejs 13.11.0
The Atom package I mentioned that might be causing this error is ide-elixir which uses ElixirLS.
If I run into this again, I will look further into my hypothesis of ElixirLS somehow causing it and bring it up there if applicable.
Same issue Guys i really don't know what to do
We want to solve this issue but like I said before we need more information from users experiencing it to find a common denominator. If you only say that it doesn't work we have nothing to go on.
We need more information to investigate this issue since it only happens for a small set of users and we currently cannot reproduce it. Are you using any non-common filesystem or disk device like a network hard drive or USB stick? What's the permissions for the deps/ directory and its subdirectories? Does it happen for all dependencies or only phoenix? Do you remember any changes you made around the time the error started happening such as upgrading any macOS/Erlang/Elixir versions?
For me i'm using Linux Debian (Kali Distro) and i'm trying to work On Phoenix Framework (elixir) But when i run mix phx.server It said that error
Just tell me what do u need to know i'll be happy to put here
@UiDesignX @ericmj I am going to try and keep this as generic as possible (not refer to OS or anything), since I gave my specific setup already, and I think I understand the source of the problem now: If 2 or more different processes try to install dependencies concurrently, one of them will run into this error, and since we are human, we usually lose to the machine.
In my case it was atom runing elixir_ls (great plugin otherwise!), so whenever I deleted deps and then ran mix deps.get
(and usually I would do something like rm deps && mix deps.get
)there were basically 2 instancces trying to install the dependencies.
Make sure there is nothing in the background that will install deps. even if you don't tell it to, and the error will probably go away.
@UiDesignX in your case it looks like the user you are running the server under does not have permissions to access certain files, can you give some specifics about what OS you are on, how you set up the project and elixir?
why is that the windows os gets the shitty support when there are more windows than linux crap...
we don't have a choice when mgt chooses the windows platform, and I wish I could use something other than the greatest web platform: phoenix, which doesn't work too well on windows. 👎
interestingly, this crap's history dates back to 2016 and no fix yet. 5 years wow!!!... Hummm,
Sorry we at Hex team don't have regular access to machines with Windows so we rely on community for support. If you are running Windows, your help with fixing compatibility issues would be appreciated. Thank you!
@ericmj
I'm a newbe at this, just started using phx, so I installed the latest versions of elixir, erlang, phoenix and postgres with tons of problems on a windows 10 insiders systems.
the versions are: elixir: 1.11.3 Phx: 1.5.7 erlang: 23 postgres:13
I then created a mix phx.new hello project
mix phx.server
connect a page to localhost:4000
then I get this miserable pile of garbage: ` File.Error could not remove file "w:/ELIXIR/PROJECTS/hello/_build/dev/lib/hello/priv": not owner Console output is shown below.
** (File.Error) could not remove file "w:/ELIXIR/PROJECTS/hello/_build/dev/lib/hello/priv": not owner (elixir 1.11.3) lib/file.ex:1131: File.rm!/1 (mix 1.11.3) lib/mix/utils.ex:439: Mix.Utils.symlink_or_copy/2 (mix 1.11.3) lib/mix/project.ex:692: Mix.Project.build_structure/2 (phoenix 1.5.7) lib/phoenix/code_reloader/server.ex:244: Phoenix.CodeReloader.Server.mix_compile/1 (phoenix 1.5.7) lib/phoenix/code_reloader/server.ex:217: Phoenix.CodeReloader.Server.mix_compile_unless_stale_config/2 (phoenix 1.5.7) lib/phoenix/code_reloader/server.ex:176: Phoenix.CodeReloader.Server.mix_compile/3 (phoenix 1.5.7) lib/phoenix/code_reloader/server.ex:75: anonymous fn/2 in Phoenix.CodeReloader.Server.handle_call/3 (phoenix 1.5.7) lib/phoenix/code_reloader/server.ex:289: Phoenix.CodeReloader.Server.proxy_io/1 (phoenix 1.5.7) lib/phoenix/code_reloader/server.ex:73: Phoenix.CodeReloader.Server.handle_call/3 (stdlib 3.8) gen_server.erl:661: :gen_server.try_handle_call/4 (stdlib 3.8) gen_server.erl:690: :gen_server.handle_msg/6 (stdlib 3.8) proc_lib.erl:249: :proc_lib.init_p_do_apply/3
`
and I don't even know how to read this crap or what it means.... so much for my first helloworld project
@wojtekmach
amazing, then don't make this available on windows so idiotic managers don't choose something that does not have any windows support. does not having a windows machine sounds logical to you? wow.... unless you're a one or two man crew, then I can see that, but Hex seems to be older that 6 or more years, so there has been plenty of time for Hex to have spend a thousand bucks to get one or more than one, hell you can get one for less that 100 now a days since every body hates windows... they are giving them away so they can migrate to the best os in the world, linux, the one and only.... phew.....
and my last comment on this is: WHY IS THE INCIDENT CLOSED SINCE 2016, WHEN USERS ARE STILL GETTING THE ERROR WITH THE LATEST VERSIONS....IN 2021
@lcr0815
Please be polite to people not getting even paid for providing us all an open-source service. Your error even is not relatet to hexpm/hex...
That's why things don't get fixed, people don't get paid, things are not related, and best of all this is open source.... wow.. what a set of great excuses, never mind the stress this creates, the irritation of not being able to do one's work simple because nobody gets paid. somebody must be making the money otherwise companies would not be sponsoring these software packages for their production especially in today's environment where everything is based on web apps and phoenix promotes itself as the best web platform... do you see where the frustration is coming from, and I still can run my phoenix helloworld app, but I can run my node helloworld app. not my choice though...
so you have two problems: an application that does not run and a job that does not pay.... so much for excuses....
now if you had a windows expert where would you start in helping to learn to debug this stuff?
10-4
Sorry, I understand. True, programming is often very frustrating...
Delete your folder called _build
and then try again...
Probably you are also running your Phoenix application on a file system that is causing the trouble (w:/ ??).
now we are getting somewhere, ? why delete the _build folder?
the next hint is how to start the debugging process?
I'm starting by looking at the function creating the problem, the elixir/lib/file.ex function file.rm what I would like to know is how to setup break points and start running the app with a breakpoint on VS.CODE
the function looks like this:
@doc """ Tries to delete the file
path`.
Returns :ok
if successful, or {:error, reason}
if an error occurs.
Note the file is deleted even if in read-only mode.
Typical error reasons are:
* `:enoent` - the file does not exist
* `:eacces` - missing permission for the file or one of its parents
* `:eperm` - the file is a directory and user is not super-user
* `:enotdir` - a component of the file name is not a directory;
on some platforms, `:enoent` is returned instead
* `:einval` - filename had an improper type, such as tuple
File.rm("file.txt")
#=> :ok
File.rm("tmp_dir/")
#=> {:error, :eperm}
""" @spec rm(Path.t()) :: :ok | {:error, posix} def rm(path) do path = IO.chardata_to_string(path)
case :file.delete(path) do
:ok ->
:ok
{:error, :eacces} = e ->
change_mode_windows(path) || e
{:error, _} = e ->
e
end
end
`
since I'm new to elixir and the rest of the lot, this means squat to me..... for now...
and if this is not the place to publish this stuff then where should I go, besides HELL
...and now for the delete _build process:
_build\dev\lib\hello - The process cannot access the file because it is being used by another process.
and the beat goes on.... and on and on...... 🥇
I would recommend you the Elixir Forum -> https://elixirforum.com
But there are other community channels as well (Discord, Slack) -> https://elixir-lang.org
@cw79
w:\ is a windows drive like c:\, windows uses a:\ - z:\ for data drives
@cw789 got me thinking and using my head is good thanks
The problem is windows assigns a SYMLINK to _build directory to point to a real directory elsewhere in the project structure. I happen to be using a substituted drive letters. W: for the project in this case, however, some apps don't like the symlink on a substituted drive, and I'm getting started on the phoenix platform, phoenix is an app that uses symlinks and it wants the actual drive letter instead of a subs letter, in my case the drive is I: and the subs letter is W:, phoenix allows me to create and run the project but comes up with a delete error message
(File.Error) could not remove file "w:/ELIXIR/PROJECTS/hello/_build/dev/lib/hello/priv": not owner
so I'm forced to use the real drive letter with a kilometric path string
substituted drive letters are created using the command subst c:\foo w: and now you can use w: as an alias for C:\foo
To my way of thinking, there is still a problem somewhere in either phoenix or windows, a delete should not fail and crap the entire application because a file can not be deleted., MO...
my problem has been resolved...
Hi @lcr0815, you are welcome to not use Elixir. This attitude is not needed in the community. Alternatively, contribute positively in future conversations and help maintainers address issues. Good luck on whatever you choose!
PS1: The issue you reported is different from the one linked here. file already exists / permission denied are caused by different reasons than not owner.
PS2: Locking because the later report was not even related to Hex. If someone runs into a similar issue, please provide steps on the elixir main repo, because I cannot reproduce this on my Windows VM.
This is posted here from https://github.com/elixir-lang/elixir/issues/5155 as per josevalim.
Precheck
Keep getting this error in a few cases, such as in the phoenix reloader this happens constantly:
as well as just got it while running
mix deps.update --all
:however I am able to manually delete the directories and can continue in all cases.
Using Process Explorer shows there are no applications holding on to those directories, thus this implies there is a race condition between elixir/mix itself doing something else and the
File.rm_rf
call.Environment
Current behavior
It is throwing a "file already exists' error when deleting a directory.
Expected behavior
File.rm_rf
should delete a directory tree without throwing a 'file already exists' error while successfully deleting the directory.