search

zephyrproject-rtos / west

West, Zephyr's meta-tool
https://docs.zephyrproject.org/latest/guides/west/index.html
Apache License 2.0
225 stars 116 forks source link

west init completes but leaves everything in zephyr/manifest-tmp #558

Closed jefffhaynes closed 12 months ago

jefffhaynes commented 2 years ago

I have no idea what causes this or how to reliably reproduce it, but running west init on windows occasionally leaves everything in the zephyr/manifest-tmp folder upon "successful" completion. west update subsequently fails because everything is in the wrong place.

marc-hb commented 2 years ago

Can you remember any error message, even vaguely?

Going forward, can you use west -v -v init?

jefffhaynes commented 2 years ago

There is no error message until I run update as the init command doesn't fail, it just leaves everything in the wrong place. I'll try the new command.

marc-hb commented 2 years ago

It could fail without a visible error message. It's extremely rare with Python and even rarer with west but it's theoretically possible. Next time this happens please use echo %errorlevel% immediately

jefffhaynes commented 2 years ago

Running west -v -v init seems to have worked, although I don't know if that was the intention or if it just happened to work that time. If it happens again I'll try to get the error level. Thanks.

mbolivar-nordic commented 2 years ago

-v -v should just increase the verbosity level without otherwise changing behavior (other than precise timing details of when things happen, of course), but the verbose logs would be very useful for diagnosing this. I haven't seen this happen myself, but I believe you, and it'd be good to get more details from the verbose logs.

congnguyenhuu commented 2 years ago

Hi everyone, I have same problem when run west init on windows. All everything in the folder manifest-tmp moved folder zephyr, but The folder manifest-tmp is not removed. That make can not run west update. This is error display, Anyone have solution ? Thanks !

west.manifest: loaded C:\Users\congnh\zephyrproject\.west\manifest-tmp\zephyrbsp.yml
Traceback (most recent call last):
  File "c:\python38\lib\shutil.py", line 780, in move
    os.rename(src, real_dst)
PermissionError: [WinError 5] Access is denied: 'C:\\Users\\congnh\\zephyrproject\\.west\\manifest-tmp' -> 'C:\\Users\\congnh\\zephyrproject\\zephyr'
marc-hb commented 2 years ago

Can you reproduce every time? If yes then please follow the instructions above.

congnguyenhuu commented 2 years ago

yes, it can reproduce. you mean use west -v -v init but I try and can not

jefffhaynes commented 2 years ago

I think he's asking you to post what it says when you do.

congnguyenhuu commented 2 years ago

This is all display when you did, it relates permission access folder when run os.rename

$ west -v -v init -m https://bitbucket.com/xx/xx/zephyr.git --mr xxxx.xx.xx.x --mf xxxx.yml
=== Initializing in C:\Users\congnh\zephyrproject
--- Cloning manifest repository from https://bitbucket.com/xx/xx/zephyr.git
running 'git clone --branch  https://bitbucket.com/'C:\Users\congnh\zephyrproject\.west\manifest-tmp'' in C:\Users\congnh\zephyrproject
Cloning into 'C:\Users\congnh\zephyrproject\.west\manifest-tmp'...
remote: Enumerating objects: 578417, done.
remote: Counting objects: 100% (578417/578417), done.
remote: Compressing objects: 100% (126962/126962), done.
remote: Total 578417 (delta 440454), reused 577231 (delta 439607)
Receiving objects: 100% (578417/578417), 383.54 MiB | 2.33 MiB/s, done.
Resolving deltas: 100% (440454/440454), done.
Note: switching to 'khsvt32h567586t8k'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

Updating files: 100% (13543/13543), done.
west.manifest: loading C:\Users\congnh\zephyrproject\.west\manifest-tmp\zephyrbsp.yml
west.manifest: group-filter: unset
west.manifest: added project cmsis path modules/hal/cmsis revision from C:\Users\congnh\zephyrproject\.west\manifest-tmp\zephyrbsp.yml
west.manifest: added project hal_nxp path modules/hal/nxp revision from C:\Users\congnh\zephyrproject\.west\manifest-tmp\zephyrbsp.yml
west.manifest: loaded C:\Users\congnh\zephyrproject\.west\manifest-tmp\zephyrbsp.yml
Traceback (most recent call last):
  File "c:\python38\lib\shutil.py", line 780, in move
    os.rename(src, real_dst)
PermissionError: [WinError 5] Access is denied: 'C:\\Users\\congnh\\zephyrproject\\.west\\manifest-tmp' -> 'C:\\Users\\congnh\\zephyrproject\\zephyr'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "c:\python38\lib\runpy.py", line 192, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "c:\python38\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Python38\Scripts\west.exe\__main__.py", line 9, in <module>
  File "c:\python38\lib\site-packages\west\app\main.py", line 781, in main
    app.run(argv or sys.argv[1:])
  File "c:\python38\lib\site-packages\west\app\main.py", line 106, in run
    self.run_command(argv)
  File "c:\python38\lib\site-packages\west\app\main.py", line 336, in run_command
    cmd.run(args, unknown, self.topdir, manifest=self.manifest)
  File "c:\python38\lib\site-packages\west\commands.py", line 140, in run
    self.do_run(args, unknown)
  File "c:\python38\lib\site-packages\west\app\project.py", line 198, in do_run
    topdir = self.bootstrap(args)
  File "c:\python38\lib\site-packages\west\app\project.py", line 305, in bootstrap
    shutil.move(os.fspath(tempdir), os.fspath(manifest_abspath))
  File "c:\python38\lib\shutil.py", line 792, in move
    rmtree(src)
  File "c:\python38\lib\shutil.py", line 730, in rmtree
    return _rmtree_unsafe(path, onerror)
  File "c:\python38\lib\shutil.py", line 603, in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)
  File "c:\python38\lib\shutil.py", line 603, in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)
  File "c:\python38\lib\shutil.py", line 603, in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)
  File "c:\python38\lib\shutil.py", line 608, in _rmtree_unsafe
    onerror(os.unlink, fullname, sys.exc_info())
  File "c:\python38\lib\shutil.py", line 606, in _rmtree_unsafe
    os.unlink(fullname)
PermissionError: [WinError 5] Access is denied: 'C:\\Users\\congnh\\zephyrproject\\.west\\manifest-tmp\\.git\\objects\\pack\\pack-2535647586gfhf56757.idx'
marc-hb commented 2 years ago

PermissionError: [WinError 5] Access is denied: 'C:\Users\congnh\zephyrproject\.west\manifest-tmp' -> 'C:\Users\congnh\zephyrproject\zephyr'

What happens if you try to rename this yourself from a command window?

west.manifest: loaded C:\Users\congnh8\Desktop\New folder.west\manifest-tmp\zephyrbsp.yml

Even stranger: is this imported from another manifest?

This is not the same problem as originally reported in the description BTW

congnguyenhuu commented 2 years ago

west.manifest: loaded C:\Users\congnh8\Desktop\New folder.west\manifest-tmp\zephyrbsp.yml sorry, I rename it , it must be this west.manifest: loaded C:\Users\congnh\zephyrproject\manifest-tmp\zephyrbsp.yml

mbolivar-nordic commented 2 years ago

sorry, I rename it , it must be this

I don't understand what you mean, sorry. It looks like you have edited the logs. This makes it hard to understand them since I cannot know for sure how the output comes from the input, or even if the output is accurate.

Maybe you are using a private repository. I understand if you cannot share that.

But do you maybe have some way to make a manifest you can share which reproduces the issue for you? That would be really helpful, thank you.

congnguyenhuu commented 2 years ago

yep. because some private information, i cannot post, so edited somethings in logs. but basically it is also same

marc-hb commented 2 years ago

If you cannot reproduce with the open-source Zephyr manifest (did you try?), it means something in your manifest(s) is triggering the problem. To identify what triggers the issue, remove the private elements one by one until the problem disappears. Now you found what triggers the issue. Finally, to reproduce the issue without any confidential information, add something similar (and not confidential) to the open source manifest. Now you have a bug report that you can fully share and something in common we can all work on.

congnguyenhuu commented 2 years ago

With community source Zephyr, I have same problem. I try with other machines, some can work but some cannt

C:\Users\datnd14>west init -m https://github.com/zephyrproject-rtos/zephyr.git --mr zephyr-v2.7.0 --mf west.yml
=== Initializing in C:\Users\datnd14
--- Cloning manifest repository from https://github.com/zephyrproject-rtos/zephyr.git, rev. zephyr-v2.7.0
Initialized empty Git repository in C:/Users/datnd14/.west/manifest-tmp/.git/
remote: Enumerating objects: 642618, done.
remote: Counting objects: 100% (2/2), done.
remote: Total 642618 (delta 1), reused 1 (delta 1), pack-reused 642616 eceiving objects: 100% (642618/642618), 4
Receiving objects: 100% (642618/642618), 433.69 MiB | 25.00 KiB/s, done.
Resolving deltas: 100% (488975/488975), done.

* [new tag] zephyr-v2.7.0 -> zephyr-v2.7.0
503c7138df2b7b0afa0f649c783b33e8d971f4d0 refs/tags/zephyr-v2.7.0
Updating files: 100% (17693/17693), done.
Note: switching to 'zephyr-v2.7.0'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

git switch -c <new-branch-name>

Or undo this operation with:

git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at 3f826560aa release: Zephyr 2.7.0
Traceback (most recent call last):
File "c:\users\datnd14\appdata\local\programs\python\python39\lib\shutil.py", line 806, in move
os.rename(src, real_dst)
PermissionError: [WinError 5] Access is denied: 'C:\\Users\\datnd14\\.west\\manifest-tmp' -> 'C:\\Users\\datnd14\\zephyr'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "c:\users\datnd14\appdata\local\programs\python\python39\lib\runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "c:\users\datnd14\appdata\local\programs\python\python39\lib\runpy.py", line 87, in _run_code
exec(code, run_globals)
File "C:\Users\datnd14\AppData\Local\Programs\Python\Python39\Scripts\west.exe\__main__.py", line 7, in <module>
File "c:\users\datnd14\appdata\local\programs\python\python39\lib\site-packages\west\app\main.py", line 779, in main
app.run(argv or sys.argv[1:])
File "c:\users\datnd14\appdata\local\programs\python\python39\lib\site-packages\west\app\main.py", line 106, in run
self.run_command(argv)
File "c:\users\datnd14\appdata\local\programs\python\python39\lib\site-packages\west\app\main.py", line 336, in run_command
cmd.run(args, unknown, self.topdir, manifest=self.manifest)
File "c:\users\datnd14\appdata\local\programs\python\python39\lib\site-packages\west\commands.py", line 132, in run
self.do_run(args, unknown)
File "c:\users\datnd14\appdata\local\programs\python\python39\lib\site-packages\west\app\project.py", line 198, in do_run
topdir = self.bootstrap(args)
File "c:\users\datnd14\appdata\local\programs\python\python39\lib\site-packages\west\app\project.py", line 295, in bootstrap
shutil.move(os.fspath(tempdir), os.fspath(manifest_abspath))
File "c:\users\datnd14\appdata\local\programs\python\python39\lib\shutil.py", line 824, in move
rmtree(src)
File "c:\users\datnd14\appdata\local\programs\python\python39\lib\shutil.py", line 740, in rmtree
return _rmtree_unsafe(path, onerror)
File "c:\users\datnd14\appdata\local\programs\python\python39\lib\shutil.py", line 613, in _rmtree_unsafe
_rmtree_unsafe(fullname, onerror)
File "c:\users\datnd14\appdata\local\programs\python\python39\lib\shutil.py", line 613, in _rmtree_unsafe
_rmtree_unsafe(fullname, onerror)
File "c:\users\datnd14\appdata\local\programs\python\python39\lib\shutil.py", line 613, in _rmtree_unsafe
_rmtree_unsafe(fullname, onerror)
File "c:\users\datnd14\appdata\local\programs\python\python39\lib\shutil.py", line 618, in _rmtree_unsafe
onerror(os.unlink, fullname, sys.exc_info())
File "c:\users\datnd14\appdata\local\programs\python\python39\lib\shutil.py", line 616, in _rmtree_unsafe
os.unlink(fullname)
PermissionError: [WinError 5] Access is denied: 'C:\\Users\\datnd14\\.west\\manifest-tmp\\.git\\objects\\pack\\pack-92b9de99850183d98ff55ccee114517db6eeae5c.idx'
marc-hb commented 2 years ago

What happens when you try this after the failure:

del C:\Users\datnd14\.west\manifest-tmp\.git\objects\pack\pack-92b9de99850183d98ff55ccee114517db6eeae5c.idx

Please also share the output of west init -v -v... and edit your comment above.

congnguyenhuu commented 2 years ago

I tried delete that, but have access error although I grant permission

del C:\Users\datnd14\.west\manifest-tmp\.git\objects\pack\pack-5fe6ccedee5ad97c70bfdbc8b9476c8c056f7a98.idx
C:\Users\datnd14\.west\manifest-tmp\.git\objects\pack\pack-5fe6ccedee5ad97c70bfdbc8b9476c8c056f7a98.idx
Access is denied.

when run west init -v -v, log is quite long, so you can read in this file log.txt

marc-hb commented 2 years ago

Thanks! Can you please try this and report any error?

cd C:\Users\datnd14\
git init w558
cd w558
git remote add origin -- https://github.com/zephyrproject-rtos/zephyr.git
git fetch origin --tags -- zephyr-v2.7.0 'refs/heads/*:refs/remotes/origin/*'
del .git\objects\pack\*
congnguyenhuu commented 2 years ago

here is result when I run that

C:\Users\datnd14\w588>git remote add origin -- https://github.com/zephyrproject-rtos/zephyr.git

C:\Users\datnd14\w588>git fetch origin --tags -- zephyr-v2.7.0 'refs/heads/*:refs/remotes/origin/*'
remote: Enumerating objects: 607149, done.
remote: Counting objects: 100% (2/2), done.
remote: Total 607149 (delta 1), reused 1 (delta 1), pack-reused 607147
Receiving objects: 100% (607149/607149), 417.88 MiB | 596.00 KiB/s, done.
Resolving deltas: 100% (461118/461118), done.
From https://github.com/zephyrproject-rtos/zephyr
* tag zephyr-v2.7.0 -> FETCH_HEAD
* [new tag] v1.0.0 -> v1.0.0
* [new tag] v1.1.0 -> v1.1.0
* [new tag] v1.1.0-rc1 -> v1.1.0-rc1
* [new tag] v1.10.0 -> v1.10.0
* [new tag] v1.10.0-rc1 -> v1.10.0-rc1
* [new tag] v1.10.0-rc2 -> v1.10.0-rc2
* [new tag] v1.10.0-rc3 -> v1.10.0-rc3

    [snip]

* [new tag] zephyr-v2.2.0 -> zephyr-v2.2.0
* [new tag] zephyr-v2.2.1 -> zephyr-v2.2.1
* [new tag] zephyr-v2.3.0 -> zephyr-v2.3.0
* [new tag] zephyr-v2.4.0 -> zephyr-v2.4.0
* [new tag] zephyr-v2.5.0 -> zephyr-v2.5.0
* [new tag] zephyr-v2.6.0 -> zephyr-v2.6.0
* [new tag] zephyr-v2.7.0 -> zephyr-v2.7.0

C:\Users\datnd14\w588>del .git\objects\pack\*
C:\Users\datnd14\w588\.git\objects\pack\*, Are you sure (Y/N)? Y
C:\Users\datnd14\w588\.git\objects\pack\pack-9554992bbd0bdd6f346a8acf37ca89822648585f.idx
Access is denied.
C:\Users\datnd14\w588\.git\objects\pack\pack-9554992bbd0bdd6f346a8acf37ca89822648585f.pack
Access is denied.
marc-hb commented 2 years ago

Great, so now we know you don't have a west problem. I think you have a problem with your git client and/or the way Windows permissions are set up in your home directory. Can you try the same thing with a USB stick or just in a different directory?

It seems normal for git to create read only pack files but somehow this is not a problem for most people. Windows permissions can be very complicated and I'm no expert https://www.ntfs.com/ntfs-permissions-file-effective.htm

congnguyenhuu commented 2 years ago

yes, I think too and I granted permissions, reinstall git with chocolatey many times. but it can not work

jaredwolff commented 2 years ago

I'm getting the same error invoking west via cmd.exe though a VSCode extension.

Full command is:

C:\Windows\System32\cmd.exe /d /c west -v -v init -m https://github.com/circuitdojo/nrf9160-feather-examples-and-drivers.git

Here's the output. 

> Executing task: west -v -v init -m https://github.com/circuitdojo/nrf9160-feather-examples-and-drivers.git <

=== Initializing in C:\Zephyr
--- Cloning manifest repository from https://github.com/circuitdojo/nrf9160-feather-examples-and-drivers.git
running 'git clone https://github.com/circuitdojo/nrf9160-feather-examples-and-drivers.git 'C:\Zephyr\.west\manifest-tmp'' in C:\Zephyr
Cloning into 'C:\Zephyr\.west\manifest-tmp'...
remote: Enumerating objects: 899, done.
remote: Counting objects: 100% (899/899), done.
remote: Compressing objects: 100% (521/521), done.
remote: Total 899 (delta 472), reused 750 (delta 327), pack-reused 0
Receiving objects: 100% (899/899), 6.71 MiB | 9.22 MiB/s, done.     
Resolving deltas: 100% (472/472), done.
west.manifest: loading C:\Zephyr\.west\manifest-tmp\west.yml
west.manifest: group-filter: unset
west.manifest: added project nrf path nrf revision v1.7.1 from C:\Zephyr\.west\manifest-tmp\west.yml
west.manifest: project <Project nrf ('C:\\Zephyr\\nrf') at v1.7.1>: ignored import (True) 
west.manifest: added project golioth path modules/lib/golioth revision main from C:\Zephyr\.west\manifest-tmp\west.yml
west.manifest: project <Project golioth ('C:\\Zephyr\\modules\\lib\\golioth') at main>: ignored import ({'name-allowlist': ['qcbor']})
west.manifest: added project pyrinas path pyrinas revision v1.7.x from C:\Zephyr\.west\manifest-tmp\west.yml
west.manifest: project <Project pyrinas ('C:\\Zephyr\\pyrinas') at v1.7.x>: ignored import (True)
west.manifest: loaded C:\Zephyr\.west\manifest-tmp\west.yml
Traceback (most recent call last):
  File "C:\Python39\lib\shutil.py", line 803, in move
    os.rename(src, real_dst)
PermissionError: [WinError 5] Access is denied: 'C:\\Zephyr\\.west\\manifest-tmp' -> 'C:\\Zephyr\\nfed'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Python39\lib\runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Python39\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Users\Jared Wolff\.zephyrtools\env\Scripts\west.exe\__main__.py", line 7, in <module>
  File "C:\Users\Jared Wolff\.zephyrtools\env\lib\site-packages\west\app\main.py", line 781, in main
    app.run(argv or sys.argv[1:])
  File "C:\Users\Jared Wolff\.zephyrtools\env\lib\site-packages\west\app\main.py", line 106, in run
    self.run_command(argv)
  File "C:\Users\Jared Wolff\.zephyrtools\env\lib\site-packages\west\app\main.py", line 336, in run_command
    cmd.run(args, unknown, self.topdir, manifest=self.manifest)
  File "C:\Users\Jared Wolff\.zephyrtools\env\lib\site-packages\west\commands.py", line 140, in run
    self.do_run(args, unknown)
  File "C:\Users\Jared Wolff\.zephyrtools\env\lib\site-packages\west\app\project.py", line 198, in do_run
    topdir = self.bootstrap(args)
  File "C:\Users\Jared Wolff\.zephyrtools\env\lib\site-packages\west\app\project.py", line 305, in bootstrap
    shutil.move(os.fspath(tempdir), os.fspath(manifest_abspath))
  File "C:\Python39\lib\shutil.py", line 815, in move
    rmtree(src)
  File "C:\Python39\lib\shutil.py", line 737, in rmtree
    return _rmtree_unsafe(path, onerror)
  File "C:\Python39\lib\shutil.py", line 610, in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)
  File "C:\Python39\lib\shutil.py", line 610, in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)
  File "C:\Python39\lib\shutil.py", line 610, in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)
  File "C:\Python39\lib\shutil.py", line 615, in _rmtree_unsafe
    onerror(os.unlink, fullname, sys.exc_info())
  File "C:\Python39\lib\shutil.py", line 613, in _rmtree_unsafe
    os.unlink(fullname)
PermissionError: [WinError 5] Access is denied: 'C:\\Zephyr\\.west\\manifest-tmp\\.git\\objects\\pack\\pack-ae179110ddff9c4a119793934f0f44c69e9e0976.idx'
The terminal process "C:\Windows\System32\cmd.exe /d /c west -v -v init -m https://github.com/circuitdojo/nrf9160-feather-examples-and-drivers.git" terminated with exit code: 1.

Terminal will be reused by tasks, press any key to close it.

Quite bizarre since west creates the files that it's copying to. Seems to be mostly related to the os.unlink function. I'm starting from a completely clean directory.

For context, I was using git to clone before but west init had an added benefit to initialize the application repo as the correct name as corresponding to self.path in west.yml

jaredwolff commented 2 years ago

Diff of the environment. Works in a stand alone cmd but not within VSCode

1d0
< !::=::\
3d1
< !ExitCode=00000000
9a8
> CHROME_CRASHPAD_PIPE_NAME=\\.\pipe\crashpad_9232_XCMYRGDVZKPZBDFN
24a24
> ORIGINAL_XDG_CURRENT_DESKTOP=undefined
26c26,27
< PATH=/cygdrive/c/Python39/Scripts:/cygdrive/c/Windows/system32:/cygdrive/c/Windows:/cygdrive/c/Windows/System32/Wbem:/cygdrive/c/Windows/System32/WindowsPowerShell/v1.0:/usr/bin:/cygdrive/c/Windows/System32/OpenSSH:/cygdrive/c/Program Files/Wireshark:/cygdrive/c/Program Files/PuTTY:/cygdrive/c/ProgramData/chocolatey/bin:/cygdrive/c/Program Files/Git/cmd:/cygdrive/c/bin:/cygdrive/c/Python39:/cygdrive/c/Program Files (x86)/Nordic Semiconductor/nrf-command-line-tools/bin:/cygdrive/c/Program Files (x86)/SEGGER/JLink:/cygdrive/c/Program Files (x86)/ZeroTier/One:/cygdrive/c/Program Files/Microsoft SQL Server/120/Tools/Binn:/cygdrive/c/Program Files/Microsoft SQL Server/Client SDK/ODBC/110/Tools/Binn:/cygdrive/c/Program Files (x86)/Microsoft SQL Server/120/Tools/Binn:/cygdrive/c/Program Files/Microsoft SQL Server/120/DTS/Binn:/cygdrive/c/Program Files (x86)/nodejs:/cygdrive/c/Users/Jared Wolff/AppData/Local/Microsoft/WindowsApps:/cygdrive/c/Users/Jared Wolff/AppData/Local/Programs/Microsoft VS Code/bin:/cygdrive/c/bin:/cygdrive/c/Python39:/cygdrive/c/Users/Jared Wolff/AppData/Roaming/npm
---
> PATH=/cygdrive/c/Python39/Scripts:/cygdrive/c/Windows/system32:/cygdrive/c/Windows:/cygdrive/c/Windows/System32/Wbem:/cygdrive/c/Windows/System32/WindowsPowerShell/v1.0:/usr/bin:/cygdrive/c/Windows/System32/OpenSSH:/cygdrive/c/Program Files/Wireshark:/cygdrive/c/Program Files/PuTTY:/cygdrive/c/ProgramData/chocolatey/bin:/cygdrive/c/Program Files/Git/cmd:/cygdrive/c/bin:/cygdrive/c/Python39:/cygdrive/c/Program Files (x86)/Nordic Semiconductor/nrf-command-line-tools/bin:/cygdrive/c/Program Files (x86)/SEGGER/JLink:/cygdrive/c/Program Files (x86)/ZeroTier/One:/cygdrive/c/Program Files/Microsoft SQL Server/120/Tools/Binn:/cygdrive/c/Program Files/Microsoft SQL Server/Client SDK/ODBC/110/Tools/Binn:/cygdrive/c/Program Files (x86)/Microsoft SQL 
> Server/120/Tools/Binn:/cygdrive/c/Program Files/Microsoft SQL Server/120/DTS/Binn:/cygdrive/c/Program Files (x86)/nodejs:/cygdrive/c/Users/Jared Wolff/AppData/Local/Microsoft/WindowsApps:/cygdrive/c/Users/Jared Wolff/AppData/Local/Programs/Microsoft VS Code/bin:/cygdrive/c/bin:/cygdrive/c/Python39:/cygdrive/c/Users/Jared Wolff/AppData/Roaming/npm
49a51,59
> TERM_PROGRAM=vscode
> TERM_PROGRAM_VERSION=1.63.2
> LANG=en_US.UTF-8
> COLORTERM=truecolor
> VSCODE_GIT_IPC_HANDLE=\\.\pipe\vscode-git-35da2a6a1e-sock
> GIT_ASKPASS=c:\Users\Jared Wolff\AppData\Local\Programs\Microsoft VS Code\resources\app\extensions\git\dist\askpass.sh
> VSCODE_GIT_ASKPASS_NODE=C:\Users\Jared Wolff\AppData\Local\Programs\Microsoft VS Code\Code.exe
> VSCODE_GIT_ASKPASS_EXTRA_ARGS=--ms-enable-electron-run-as-node
> VSCODE_GIT_ASKPASS_MAIN=c:\Users\Jared Wolff\AppData\Local\Programs\Microsoft VS Code\resources\app\extensions\git\dist\askpass-main.js

Cygwin looks like the culprit

jaredwolff commented 2 years ago

Looks like it's because I'm running west in a virtual environment. Without the virtual environment, works ok. With the path to the virtual environment set, it breaks.

The plot thickens.

After lots of testing. I've narrowed it down to whether or not VSCode has the folder open. It appears to be scanning the files inside the .west folder. When west is ready to transfer the repo to it's final resting place, it's prevented from doing so because VSCode is indexing the files.

When I close VSCode and run it in a terminal with same environment, it runs perfectly fine.

Hope this helps people that may come across the same problem!

mbolivar-nordic commented 2 years ago

After lots of testing. I've narrowed it down to whether or not VSCode has the folder open. It appears to be scanning the files inside the .west folder. When west is ready to transfer the repo to it's final resting place, it's prevented from doing so because VSCode is indexing the files.

@wbober @trond-snekvik I was thinking about the VS code extension and I wanted to let you know about this to see if you've ever observed random issues like this.

Some of the above discussion looks like random permissions problems that unfortunately I have no idea how to triage right now, but this particular issue reported by @jaredwolff might be something you are running into more often.

marc-hb commented 2 years ago

it's prevented from doing so because VSCode is indexing the files.

It would help if Windows could delete files in use like Linux filesystems can (it would also save a lot of Windows reboots, I digress)

However similar issues could happen on Linux too: a few years ago, it came as a big surprise to me that some operations like "git diff" appear to be read-only but in fact they have some side effects because they can cause git to update some of its internal data: https://public-inbox.org/git/20160331140515.GA31116@sigill.intra.peff.net/ So in theory we could have a similar race where some Linux IDE triggers some not-so-read-only git commands on a half-moved repo and ends up corrupting it. Very unlikely but possible?

So, @mbolivar-nordic should west switch to some OS-dependent and "standard" /tmp locations to escape this sort of concurrency? Even if the interactions with IDEs ends up being harmless, it's not "nice" to confuse them.

mbolivar-nordic commented 2 years ago

So, @mbolivar-nordic should west switch to some OS-dependent and "standard" /tmp locations to escape this sort of concurrency? Even if the interactions with IDEs ends up being harmless, it's not "nice" to confuse them.

I thought about that, but then you're inviting issues with cross-filesystem moves, which are a can of worms of their own. If you have a strategy for accomplishing this ask, I'm open to ideas, though.

marc-hb commented 2 years ago

but then you're inviting issues with cross-filesystem moves,

OMG, I referenced something from 2016 but now you're bringing back memories from 2010!! https://bugs.eclipse.org/bugs/show_bug.cgi?id=319233 "Cloning repository fails if destination is on network drive because File.renameTo doesn't succeed"

How about zephyr/.manifest-tmp instead of zephyr/manifest-tmp? Or some "hidden" Windows file attribute? Both? @jaredwolff would you be able to test these ~on Windows~ with VS code?

jaredwolff commented 2 years ago

My work around with this, from what I remember, was disabling the Git functionality in VSCode temporarily while west did its thing. It was trying to cache or open the repository in the manifest-tmp folder and thus locking up the files. Next time I'm working on my VSCode plugin I'll have to see if changing the manifest-tmp directory name fixes it.

mbolivar-nordic commented 2 years ago

How about zephyr/.manifest-tmp instead of zephyr/manifest-tmp? Or some "hidden" Windows file attribute? Both?

I don't know what you mean, sorry. The temporary manifest repository already goes under .west, not zephyr. I don't know a lot about hidden files on windows or what vs code does. If any body has any resources they can point me at, I'd be grateful.

marc-hb commented 2 years ago

The temporary manifest repository already goes under .west, not zephyr

Sorry I blindly trusted the description and title at the very top. At the moment they are:

west init completes but leaves everything in zephyr/manifest-tmp

but running west init on windows occasionally leaves everything in the zephyr/manifest-tmp folder upon "successful" completion

marc-hb commented 2 years ago

In a totally unrelated discussion I just heard that VS2022 is automatically generating some c/Users/me/.gitconfig file. Sounds nasty, what else does it do?

jefffhaynes commented 1 year ago

I just rebuilt a new machine and this is still happening. VSCode was not running at any point. Running west -v -v init results in:

PS C:\Users\Jeff\Source\Repos\Test> west -v -v init
=== Initializing in C:\Users\Jeff\Source\Repos\Test
--- Cloning manifest repository from https://github.com/zephyrproject-rtos/zephyr
running 'git clone https://github.com/zephyrproject-rtos/zephyr 'C:\Users\Jeff\Source\Repos\Test\.west\manifest-tmp'' in C:\Users\Jeff\Source\Repos\Test
Cloning into 'C:\Users\Jeff\Source\Repos\Test\.west\manifest-tmp'...
remote: Enumerating objects: 765677, done.
remote: Counting objects: 100% (2/2), done.
remote: Total 765677 (delta 1), reused 1 (delta 1), pack-reused 765675
Receiving objects: 100% (765677/765677), 534.13 MiB | 931.00 KiB/s, done.

Resolving deltas: 100% (582305/582305), done.
Updating files: 100% (21810/21810), done.
west.manifest: loading data (no file)
west.manifest: resolving self import submanifests
west.manifest: ignored self import
west.manifest: resolved self import
west.manifest: group-filter: ['-ci']
west.manifest: added project canopennode path modules/lib/canopennode revision 53d3415c14d60f8f4bfca54bfbc5d5a667d7e724
west.manifest: added project chre path modules/lib/chre revision ef76d3456db07e4959df555047d6962279528c8d
west.manifest: added project cmsis path modules/hal/cmsis revision 093de61c2a7d12dc9253daf8692f61f793a9254a groups ['hal']
west.manifest: added project edtt path tools/edtt revision 1ea61a390d2bfcf3b2ecdba8f8b0b98dfdffbd11 groups ['tools']
west.manifest: added project fatfs path modules/fs/fatfs revision a30531af3a95a9a3ea7d771ea8a578ebfed45514 groups ['fs']
west.manifest: added project fff path modules/lib/fff revision 6ce5ba26486e93d5b7696a3e23f0585932c14b16 groups ['ci']
west.manifest: added project hal_altera path modules/hal/altera revision 0d225ddd314379b32355a00fb669eacf911e750d groups ['hal']
west.manifest: added project hal_atmel path modules/hal/atmel revision 1d237f2e2f262751975b6da6e03af569b2b49b2b groups ['hal']
west.manifest: added project hal_espressif path modules/hal/espressif revision 1e29548e4492807eb18cbaa77d3e78801cc7d07e groups ['hal']
west.manifest: added project hal_gigadevice path modules/hal/gigadevice revision dd0e0322474462b58059e6fedaf1d67d2a0864d0 groups ['hal']
west.manifest: added project hal_infineon path modules/hal/infineon revision 4af06965f57ba1e7d170e6a97d24c33785543a8c groups ['hal']
west.manifest: added project hal_microchip path modules/hal/microchip revision 5d079f1683a00b801373bbbbf5d181d4e33b30d5 groups ['hal']
west.manifest: added project hal_nordic path modules/hal/nordic revision 249199ec5a5c31d170659921048764e96d05cc0e groups ['hal']
west.manifest: added project hal_nuvoton path modules/hal/nuvoton revision b4d31f33238713a568e23618845702fadd67386f groups ['hal']
west.manifest: added project hal_nxp path modules/hal/nxp revision 3516aa5f722eb70b969b3bc57bcf9389ad901057 groups ['hal']
west.manifest: added project hal_openisa path modules/hal/openisa revision 40d049f69c50b58ea20473bee14cf93f518bf262 groups ['hal']
west.manifest: added project hal_quicklogic path modules/hal/quicklogic revision b3a66fe6d04d87fd1533a5c8de51d0599fcd08d0 groups ['hal']
west.manifest: added project hal_renesas path modules/hal/renesas revision 468d3f2146d18c7f86a4640fc641cc1d20a4a100 groups ['hal']
west.manifest: added project hal_rpi_pico path modules/hal/rpi_pico revision 191f5ba46fda49523cdaaef27583d1c875ba2c36 groups ['hal']
west.manifest: added project hal_silabs path modules/hal/silabs revision 1ec8dd99aa4ac3e8632d2aa28a7438049bb27102 groups ['hal']
west.manifest: added project hal_st path modules/hal/st revision cccbc24c14decfd3f93959f7b14514536af973c7 groups ['hal']
west.manifest: added project hal_stm32 path modules/hal/stm32 revision 5b1610c3cf46589d50e2634ba240b7b38bc59c53 groups ['hal']
west.manifest: added project hal_telink path modules/hal/telink revision 38573af589173259801ae6c2b34b7d4c9e626746 groups ['hal']
west.manifest: added project hal_ti path modules/hal/ti revision 29af528900d03af02a17b4b7f4fc93f19a81da2a groups ['hal']
west.manifest: added project hal_wurthelektronik path modules/hal/wurthelektronik revision 24ca9873c3d608fad1fea0431836bc8f144c132e groups ['hal']
west.manifest: added project hal_xtensa path modules/hal/xtensa revision 63f655362423aa49507da7977a2d37142e8debeb groups ['hal']
west.manifest: added project libmetal path modules/hal/libmetal revision 2f586b4f1276fb075ee145421bdf6cbe5403aa41 groups ['hal']
west.manifest: added project liblc3 path modules/lib/liblc3 revision 448f3de31f49a838988a162ef1e23a89ddf2d2ed
west.manifest: added project littlefs path modules/fs/littlefs revision 652f2c5646e79b881e6f3099686ad3b7af9e216c groups ['fs']
west.manifest: added project loramac-node path modules/lib/loramac-node revision 0257b50905695192d095667b1c3abb80346db1a1
west.manifest: added project lvgl path modules/lib/gui/lvgl revision af95bdfcf6784edd958ea08139c713e2d3dee7af
west.manifest: added project lz4 path modules/lib/lz4 revision 8e303c264fc21c2116dc612658003a22e933124d
west.manifest: added project mbedtls path modules/crypto/mbedtls revision 7fed49c9b9f983ad6416986661ef637459723bcb groups ['crypto']
west.manifest: added project mcuboot path bootloader/mcuboot revision 21e56a1bd033af263e808940779a1adaf4d5465e
west.manifest: added project mipi-sys-t path modules/debug/mipi-sys-t revision 0d521d8055f3b2b4842f728b0365d3f0ece9c37f groups ['debug']
west.manifest: added project nanopb path modules/lib/nanopb revision dc4deed54fd4c7e1935e3b6387eedf21bb45dc38
west.manifest: added project net-tools path tools/net-tools revision e0828aa9629b533644dc96ff6d1295c939bd713c groups ['tools']
west.manifest: added project nrf_hw_models path modules/bsim_hw_models/nrf_hw_models revision 65bc5305d432c08e24a3f343006d1e7deaff4908
west.manifest: added project open-amp path modules/lib/open-amp revision 8d53544871e1f300c478224faca6be8384ab0d04
west.manifest: added project openthread path modules/lib/openthread revision b21e99b4b3d823f71c902b9174ff62b964c124f0
west.manifest: added project picolibc path modules/lib/picolibc revision 04ada5951cbaf8e7b17f8226ce31cb6837c28ba7
west.manifest: added project segger path modules/debug/segger revision d4e568a920b4bd087886170a5624c167b2d0665e groups ['debug']
west.manifest: added project sof path modules/audio/sof revision fed466c264ad078c66f7bff9218ba1d3fa0eb201
west.manifest: added project tflite-micro path modules/lib/tflite-micro revision 9156d050927012da87079064db59d07f03b8baf6
west.manifest: added project tinycbor path modules/lib/tinycbor revision 9e1f34bc08123aaad7666d3652aaa839e8178b3b
west.manifest: added project tinycrypt path modules/crypto/tinycrypt revision 3e9a49d2672ec01435ffbf0d788db6d95ef28de0 groups ['crypto']
west.manifest: added project TraceRecorderSource path modules/debug/TraceRecorder revision 9893bf1cf649a2c4ee2e27293f887994f3d0da5b groups ['debug']
west.manifest: added project trusted-firmware-m path modules/tee/tf-m/trusted-firmware-m revision 4c984817d861512edaeca27b83308e9b9915a98a groups ['tee']
west.manifest: added project tf-m-tests path modules/tee/tf-m/tf-m-tests revision c99a86b295c4887520da9d8402566d7f225c974e groups ['tee']
west.manifest: added project psa-arch-tests path modules/tee/tf-m/psa-arch-tests revision f4fc2442b8e29e2a03d9899e46e5a3ea3df8c2c9 groups ['tee']
west.manifest: added project uoscore-uedhoc path modules/lib/uoscore-uedhoc revision e8920192b66db4f909eb9cd3f155d5245c1ae825
west.manifest: added project zcbor path modules/lib/zcbor revision a0d6981f14d4001d6f0d608d1a427f9bc6bb6d02
west.manifest: added project zscilib path modules/lib/zscilib revision a54986aa98db4082ac56b582843bb5b5435208a6
west.manifest: loaded data (no file)
FATAL ERROR: Destination path 'C:\Users\Jeff\Source\Repos\Test\zephyr\manifest-tmp' already exists

After running the files once again appear to be in .west/manifest-tmp and at this point I can't init or update. It would be great if we could get a bug label on this. This is a pristine copy of Windows 11 and I've followed the Zephyr setup instructions to a T. There is obviously an issue of some sort.

jefffhaynes commented 1 year ago

Ok I misread the error and it looks like I still had a zephyr/manifest-tmp directory, possibly left over from the "vs code" issue. Still, the whole process seems quite fragile.

marc-hb commented 1 year ago

If you don't wipe out C:\Users\Jeff\Source\Repos\Test to actually start over then many things can happen; I wouldn't call that "fragile".

jefffhaynes commented 1 year ago

I have a repo at that location so I can't wipe it out. Checking out zephyr to a different project and then running the environment scripts has never worked for me so I have zephyr in my project directly, the same way many projects do.

marc-hb commented 1 year ago

Understood. Makes sense. You should at least delete .west/ completely:

jefffhaynes commented 1 year ago

I had but I missed the zephyr directory, thanks

timonsku commented 1 year ago

Had something related to this general issue happen too today. I used west init and it failed when it tried to link the cloned manifest (using defaults, it freshly cloned). Python threw a permission error when trying to link the manifest directory to the zephyr directory PermissionError: [WinError 5] Access is denied Unfortunately I lost the log history after executing another command. Bottom line though is that it worked fine after I deleted the .west and zephyr folder and just ran west init again. So possible that the issue came from an issue during cloning. The project was definitely "clean" before attempting the first init and had no old files laying around.

attie-argentum commented 1 year ago

I can reliably reproduce this on Linux with the following steps:

mkdir ./zephyr/
west init ./ -m https://github.com/zephyrproject-rtos/zephyr.git
Console Output ``` user@c3a266ec741a:~/zephyrproject$ west --version West version: v1.1.0 user@c3a266ec741a:~/zephyrproject$ mkdir ./zephyr/ user@c3a266ec741a:~/zephyrproject$ west -vvvv init ./ -m https://github.com/zephyrproject-rtos/zephyr.git args namespace: Namespace(help=None, zephyr_base=None, verbose=4, command='init', manifest_url='https://github.com/zephyrproject-rtos/zephyr.git', manifest_rev=None, manifest_file=None, local=False, directory='./') === Initializing in /home/user/zephyrproject --- Cloning manifest repository from https://github.com/zephyrproject-rtos/zephyr.git running 'git clone https://github.com/zephyrproject-rtos/zephyr.git /home/user/zephyrproject/.west/manifest-tmp' in /home/user/zephyrproject Cloning into '/home/user/zephyrproject/.west/manifest-tmp'... remote: Enumerating objects: 903446, done. remote: Counting objects: 100% (17022/17022), done. remote: Compressing objects: 100% (891/891), done. remote: Total 903446 (delta 16325), reused 16132 (delta 16131), pack-reused 886424 Receiving objects: 100% (903446/903446), 584.58 MiB | 14.69 MiB/s, done. Resolving deltas: 100% (685969/685969), done. west.manifest: DEBUG: loading data (no file) west.manifest: DEBUG: resolving self import submanifests west.manifest: DEBUG: ignored self import west.manifest: DEBUG: resolved self import west.manifest: DEBUG: group-filter: ['-babblesim'] west.manifest: DEBUG: added project acpica path modules/lib/acpica revision f16a0b4d0f0edd7b78a332fcf507be2187fac21e west.manifest: DEBUG: added project bsim path tools/bsim revision 384a091445c57b44ac8cbd18ebd245b47c71db94 groups ['babblesim'] west.manifest: DEBUG: added project babblesim_base path tools/bsim/components revision 19d62424c0802c6c9fc15528febe666e40f372a1 groups ['babblesim'] west.manifest: DEBUG: added project babblesim_ext_2G4_libPhyComv1 path tools/bsim/components/ext_2G4_libPhyComv1 revision 9018113a362fa6c9e8f4b9cab9e5a8f12cc46b94 groups ['babblesim'] west.manifest: DEBUG: added project babblesim_ext_2G4_phy_v1 path tools/bsim/components/ext_2G4_phy_v1 revision d47c6dd90035b41b14f6921785ccb7b8484868e2 groups ['babblesim'] west.manifest: DEBUG: added project babblesim_ext_2G4_channel_NtNcable path tools/bsim/components/ext_2G4_channel_NtNcable revision 20a38c997f507b0aa53817aab3d73a462fff7af1 groups ['babblesim'] west.manifest: DEBUG: added project babblesim_ext_2G4_channel_multiatt path tools/bsim/components/ext_2G4_channel_multiatt revision bde72a57384dde7a4310bcf3843469401be93074 groups ['babblesim'] west.manifest: DEBUG: added project babblesim_ext_2G4_modem_magic path tools/bsim/components/ext_2G4_modem_magic revision cb70771794f0bf6f262aa474848611c68ae8f1ed groups ['babblesim'] west.manifest: DEBUG: added project babblesim_ext_2G4_modem_BLE_simple path tools/bsim/components/ext_2G4_modem_BLE_simple revision 809ab073159c9ab6686c2fea5749b0702e0909f7 groups ['babblesim'] west.manifest: DEBUG: added project babblesim_ext_2G4_device_burst_interferer path tools/bsim/components/ext_2G4_device_burst_interferer revision 5b5339351d6e6a2368c686c734dc8b2fc65698fc groups ['babblesim'] west.manifest: DEBUG: added project babblesim_ext_2G4_device_WLAN_actmod path tools/bsim/components/ext_2G4_device_WLAN_actmod revision 9cb6d8e72695f6b785e57443f0629a18069d6ce4 groups ['babblesim'] west.manifest: DEBUG: added project babblesim_ext_2G4_device_playback path tools/bsim/components/ext_2G4_device_playback revision 85c645929cf1ce995d8537107d9dcbd12ed64036 groups ['babblesim'] west.manifest: DEBUG: added project babblesim_ext_libCryptov1 path tools/bsim/components/ext_libCryptov1 revision eed6d7038e839153e340bd333bc43541cb90ba64 groups ['babblesim'] west.manifest: DEBUG: added project canopennode path modules/lib/canopennode revision dec12fa3f0d790cafa8414a4c2930ea71ab72ffd west.manifest: DEBUG: added project chre path modules/lib/chre revision b7955c27e50485b7dafdc3888d7d6afdc2ac6d96 west.manifest: DEBUG: added project cmsis path modules/hal/cmsis revision 74981bf893e8b10931464b9945e2143d99a3f0a3 groups ['hal'] west.manifest: DEBUG: added project edtt path tools/edtt revision 64e5105ad82390164fb73fc654be3f73a608209a groups ['tools'] west.manifest: DEBUG: added project fatfs path modules/fs/fatfs revision 427159bf95ea49b7680facffaa29ad506b42709b groups ['fs'] west.manifest: DEBUG: added project hal_altera path modules/hal/altera revision 0d225ddd314379b32355a00fb669eacf911e750d groups ['hal'] west.manifest: DEBUG: added project hal_atmel path modules/hal/atmel revision 5ab43007eda3f380c125f957f03638d2e8d1144d groups ['hal'] west.manifest: DEBUG: added project hal_espressif path modules/hal/espressif revision ae1bd648a1ac701672c46b6ff4eadfa716937ce3 groups ['hal'] west.manifest: DEBUG: added project hal_ethos_u path modules/hal/ethos_u revision 90ada2ea5681b2a2722a10d2898eac34c2510791 groups ['hal'] west.manifest: DEBUG: added project hal_gigadevice path modules/hal/gigadevice revision 2994b7dde8b0b0fa9b9c0ccb13474b6a486cddc3 groups ['hal'] west.manifest: DEBUG: added project hal_infineon path modules/hal/infineon revision 0bebc14d8bd1a249ee7fbc70b37db6f01f72544f groups ['hal'] west.manifest: DEBUG: added project hal_intel path modules/hal/intel revision 89a0db5d9155484624447d62a5211bcf6f6f0387 groups ['hal'] west.manifest: DEBUG: added project hal_microchip path modules/hal/microchip revision 5d079f1683a00b801373bbbbf5d181d4e33b30d5 groups ['hal'] west.manifest: DEBUG: added project hal_nordic path modules/hal/nordic revision cf6e9fc5f7c2c98df26f2a4227a95df9a50823e7 groups ['hal'] west.manifest: DEBUG: added project hal_nuvoton path modules/hal/nuvoton revision 3e0a4c4d3328b2f72b164219add19d5308b53cb5 groups ['hal'] west.manifest: DEBUG: added project hal_nxp path modules/hal/nxp revision 674aa3834a60720adab03b2ba70c8b4db90f4916 groups ['hal'] west.manifest: DEBUG: added project hal_openisa path modules/hal/openisa revision d1e61c0c654d8ca9e73d27fca3a7eb3b7881cb6a groups ['hal'] west.manifest: DEBUG: added project hal_quicklogic path modules/hal/quicklogic revision b3a66fe6d04d87fd1533a5c8de51d0599fcd08d0 groups ['hal'] west.manifest: DEBUG: added project hal_renesas path modules/hal/renesas revision 346e87d5425f3a12ec8aed87cfc14d949d5ce8c7 groups ['hal'] west.manifest: DEBUG: added project hal_rpi_pico path modules/hal/rpi_pico revision b7801e4db6a62ea2d37bbef7880c3d056530c9bf groups ['hal'] west.manifest: DEBUG: added project hal_silabs path modules/hal/silabs revision fbb47a4067c460a2c26917ff34189fb4f32f654f groups ['hal'] west.manifest: DEBUG: added project hal_st path modules/hal/st revision 5948f7b3304f1628a45ee928cd607619a7f53bbb groups ['hal'] west.manifest: DEBUG: added project hal_stm32 path modules/hal/stm32 revision 1bc72c299d0365c0ee2575a97918b22df0899e10 groups ['hal'] west.manifest: DEBUG: added project hal_telink path modules/hal/telink revision 38573af589173259801ae6c2b34b7d4c9e626746 groups ['hal'] west.manifest: DEBUG: added project hal_ti path modules/hal/ti revision ae1db23f32dde779cdfc4afaa9a60ea219310a64 groups ['hal'] west.manifest: DEBUG: added project hal_wurthelektronik path modules/hal/wurthelektronik revision 24ca9873c3d608fad1fea0431836bc8f144c132e groups ['hal'] west.manifest: DEBUG: added project hal_xtensa path modules/hal/xtensa revision 41a631d4aeeeaedc0daece21eecc338807296ad7 groups ['hal'] west.manifest: DEBUG: added project libmetal path modules/hal/libmetal revision b91611a6f47dd29fb24c46e5621e797557f80ec6 groups ['hal'] west.manifest: DEBUG: added project liblc3 path modules/lib/liblc3 revision 448f3de31f49a838988a162ef1e23a89ddf2d2ed west.manifest: DEBUG: added project littlefs path modules/fs/littlefs revision ca583fd297ceb48bced3c2548600dc615d67af24 groups ['fs'] west.manifest: DEBUG: added project loramac-node path modules/lib/loramac-node revision ce57712f3e426bbbb13acaec97b45369f716f43a west.manifest: DEBUG: added project lvgl path modules/lib/gui/lvgl revision 5da257f782a8f9c6e265bdc60ebc2a93fdee24de west.manifest: DEBUG: added project lz4 path modules/lib/lz4 revision 8e303c264fc21c2116dc612658003a22e933124d west.manifest: DEBUG: added project mbedtls path modules/crypto/mbedtls revision 6e7841e5a08eb5da3c82dbc8b6b6d82ae4b7d2a0 groups ['crypto'] west.manifest: DEBUG: added project mcuboot path bootloader/mcuboot revision 76d19b3b8885ea7ae25a6f4f5d8501f7ec646447 west.manifest: DEBUG: added project mipi-sys-t path modules/debug/mipi-sys-t revision a819419603a2dfcb47f7f39092e1bc112e45d1ef groups ['debug'] west.manifest: DEBUG: added project nanopb path modules/lib/nanopb revision 42fa8b211e946b90b9d968523fce7b1cfe27617e west.manifest: DEBUG: added project net-tools path tools/net-tools revision e0828aa9629b533644dc96ff6d1295c939bd713c groups ['tools'] west.manifest: DEBUG: added project nrf_hw_models path modules/bsim_hw_models/nrf_hw_models revision 23bc852e3e4eaf64caa415b47e365ac1685ed91d west.manifest: DEBUG: added project open-amp path modules/lib/open-amp revision 42b7c577714b8f22ce82a901e19c1814af4609a8 west.manifest: DEBUG: added project openthread path modules/lib/openthread revision f7690fe7e9d638341921808cba6a3e695ec0131e west.manifest: DEBUG: added project picolibc path modules/lib/picolibc revision d07c38ff051386f8e09a143ea0a6c1d6d66dd1d8 west.manifest: DEBUG: added project segger path modules/debug/segger revision 5792675a2470d0f3857de1e77bff57b38c28de3b groups ['debug'] west.manifest: DEBUG: added project sof path modules/audio/sof revision ffbf9c2a6ea2930b0ac7e3a37c7cd7f5c417d090 west.manifest: DEBUG: added project tflite-micro path modules/lib/tflite-micro revision 9156d050927012da87079064db59d07f03b8baf6 west.manifest: DEBUG: added project tinycrypt path modules/crypto/tinycrypt revision 3e9a49d2672ec01435ffbf0d788db6d95ef28de0 groups ['crypto'] west.manifest: DEBUG: added project TraceRecorderSource path modules/debug/TraceRecorder revision 1ede7ce8e7ab226e9a84d95266241ba3f38d91d2 groups ['debug'] west.manifest: DEBUG: added project trusted-firmware-m path modules/tee/tf-m/trusted-firmware-m revision 79a6115d3a8d0e04864ae8156c1dc8532b750f5a groups ['tee'] west.manifest: DEBUG: added project trusted-firmware-a path modules/tee/tf-a/trusted-firmware-a revision 421dc050278287839f5c70019bd6aec617f2bbdb groups ['tee'] west.manifest: DEBUG: added project tf-m-tests path modules/tee/tf-m/tf-m-tests revision 0f80a65193ddbbe3f0ac38b33b07b26138c11fa7 groups ['tee'] west.manifest: DEBUG: added project psa-arch-tests path modules/tee/tf-m/psa-arch-tests revision 6a17330e0dfb5f319730f974d5b05f7b7f04757b groups ['tee'] west.manifest: DEBUG: added project uoscore-uedhoc path modules/lib/uoscore-uedhoc revision 5fe2cb613bd7e4590bd1b00c2adf181ac0229379 west.manifest: DEBUG: added project zcbor path modules/lib/zcbor revision 67fd8bb88d3136738661fa8bb5f9989103f4599e west.manifest: DEBUG: added project zscilib path modules/lib/zscilib revision 34c3432e81085bb717e4871d21ca419ae0058ec5 west.manifest: DEBUG: added project thrift path modules/lib/thrift revision 10023645a0e6cb7ce23fcd7fd3dbac9f18df6234 west.manifest: DEBUG: group filters in precedence order (later is higher): deque([['-babblesim']]) west.manifest: DEBUG: final top level group-filter: ['-babblesim'] west.manifest: DEBUG: loaded data (no file) moving /home/user/zephyrproject/.west/manifest-tmp to /home/user/zephyrproject/zephyr --- setting manifest.path to zephyr === Initialized. Now run "west update" inside /home/user/zephyrproject. user@c3a266ec741a:~/zephyrproject$ ls -la total 16 drwxrwxr-x 4 user user 4096 Aug 3 22:48 . drwxr-xr-x 6 user user 4096 Aug 3 22:43 .. drwxr-xr-x 2 user user 4096 Aug 3 22:49 .west drwxr-xr-x 3 user user 4096 Aug 3 22:49 zephyr user@c3a266ec741a:~/zephyrproject$ ls -la zephyr/ total 12 drwxr-xr-x 3 user user 4096 Aug 3 22:49 . drwxrwxr-x 4 user user 4096 Aug 3 22:48 .. drwxr-xr-x 23 user user 4096 Aug 3 22:49 manifest-tmp user@c3a266ec741a:~/zephyrproject$ ls -la zephyr/manifest-tmp/ total 424 drwxr-xr-x 23 user user 4096 Aug 3 22:49 . drwxr-xr-x 3 user user 4096 Aug 3 22:49 .. -rw-r--r-- 1 user user 765 Aug 3 22:49 .checkpatch.conf -rw-r--r-- 1 user user 2667 Aug 3 22:49 .clang-format -rw-r--r-- 1 user user 448 Aug 3 22:49 .codecov.yml -rw-r--r-- 1 user user 1213 Aug 3 22:49 .editorconfig drwxr-xr-x 8 user user 4096 Aug 3 22:49 .git -rw-r--r-- 1 user user 352 Aug 3 22:49 .gitattributes drwxr-xr-x 4 user user 4096 Aug 3 22:49 .github -rw-r--r-- 1 user user 987 Aug 3 22:49 .gitignore -rw-r--r-- 1 user user 2063 Aug 3 22:49 .gitlint -rw-r--r-- 1 user user 6276 Aug 3 22:49 .mailmap -rw-r--r-- 1 user user 266 Aug 3 22:49 .yamllint -rw-r--r-- 1 user user 76850 Aug 3 22:49 CMakeLists.txt -rw-r--r-- 1 user user 53820 Aug 3 22:49 CODEOWNERS -rw-r--r-- 1 user user 3447 Aug 3 22:49 CODE_OF_CONDUCT.md -rw-r--r-- 1 user user 1691 Aug 3 22:49 CONTRIBUTING.rst -rw-r--r-- 1 user user 187 Aug 3 22:49 Kconfig -rw-r--r-- 1 user user 29347 Aug 3 22:49 Kconfig.zephyr -rw-r--r-- 1 user user 11358 Aug 3 22:49 LICENSE -rw-r--r-- 1 user user 69169 Aug 3 22:49 MAINTAINERS.yml -rw-r--r-- 1 user user 4344 Aug 3 22:49 README.rst -rw-r--r-- 1 user user 85 Aug 3 22:49 VERSION drwxr-xr-x 13 user user 4096 Aug 3 22:49 arch drwxr-xr-x 14 user user 4096 Aug 3 22:49 boards drwxr-xr-x 17 user user 4096 Aug 3 22:49 cmake drwxr-xr-x 21 user user 4096 Aug 3 22:49 doc drwxr-xr-x 78 user user 4096 Aug 3 22:49 drivers drwxr-xr-x 13 user user 4096 Aug 3 22:49 dts drwxr-xr-x 3 user user 4096 Aug 3 22:49 include drwxr-xr-x 4 user user 4096 Aug 3 22:49 kernel drwxr-xr-x 11 user user 4096 Aug 3 22:49 lib drwxr-xr-x 3 user user 4096 Aug 3 22:49 misc drwxr-xr-x 26 user user 4096 Aug 3 22:49 modules drwxr-xr-x 23 user user 4096 Aug 3 22:49 samples drwxr-xr-x 24 user user 4096 Aug 3 22:49 scripts drwxr-xr-x 5 user user 4096 Aug 3 22:49 share drwxr-xr-x 3 user user 4096 Aug 3 22:49 snippets drwxr-xr-x 12 user user 4096 Aug 3 22:49 soc drwxr-xr-x 2 user user 4096 Aug 3 22:49 submanifests drwxr-xr-x 39 user user 4096 Aug 3 22:49 subsys drwxr-xr-x 21 user user 4096 Aug 3 22:49 tests -rw-r--r-- 1 user user 872 Aug 3 22:49 version.h.in -rw-r--r-- 1 user user 11520 Aug 3 22:49 west.yml -rw-r--r-- 1 user user 110 Aug 3 22:49 zephyr-env.cmd -rw-r--r-- 1 user user 2053 Aug 3 22:49 zephyr-env.sh ```

I know, you're not supposed to run west init in a directory that isn't empty, but it's an easy mistake to make, and no errors are reported... For example, what if you clone Zephyr and then run west init because you're new to this, or just weren't thinking straight.

I'm reading into it / making some assumptions... but in this case the issue appears to be that the target ./zephyr/ directory isn't empty - so after west init clones into ./.west/manifest-tmp/, it performs a "dumb" mv ./.west/manifest-tmp/ ./zephyr/, which results placing ./.west/manifest-tmp/ into the existing ./zephyr/ directory (final path ./zephyr/manifest-tmp/), not renaming ./.west/manifest-tmp/ to ./zephyr/... i.e: no check is performed for ./zephyr/ existing already.

This matches the behavior of /bin/mv's second mode of invocation, where perhaps the first was assumed:

SYNOPSIS
       mv [OPTION]... [-T] SOURCE DEST
       mv [OPTION]... SOURCE... DIRECTORY
       mv [OPTION]... -t DIRECTORY SOURCE...

Other modules are cloned directly into their final destination, so git is responsible for checking the state of things.

This behavior is present back to at least v0.12.0.

Understood. Makes sense. You should at least delete .west/ completely:

This isn't enough - as described here, leaving the ./zephyr/ directory will directly cause this issue (though it may not be the only way).

attie-argentum commented 1 year ago

PS: west uses shutil.move() to perform this move, with no other checks.

If the destination is an existing directory, then src is moved inside that directory. If the destination already exists but is not a directory, it may be overwritten depending on os.rename() semantics.

https://github.com/zephyrproject-rtos/west/blob/9e8f5002f2ee86b7150c039cb69aa653ff454162/src/west/app/project.py#L324-L332

attie-argentum commented 1 year ago

I believe that the issue reported by @congnguyenhuu / @jaredwolff / @timonsku is different to the original issue, and is certainly different to mine.

The crux is:

PermissionError: [WinError 5] Access is denied: 'C:\\Users\\congnh\\zephyrproject\\.west\\manifest-tmp' -> 'C:\\Users\\congnh\\zephyrproject\\zephyr'

@jaredwolff may be on to something with this comment:

After lots of testing. I've narrowed it down to whether or not VSCode has the folder open. It appears to be scanning the files inside the .west folder. When west is ready to transfer the repo to it's final resting place, it's prevented from doing so because VSCode is indexing the files.

... if any of you are still experiencing this, please could you open a new ticket to discuss further? (or perhaps this issue should be renamed accordingly, and I should remove the 'Fixes' tag from my PR)

jaredwolff commented 1 year ago

@attie-argentum the issue, in my case, seemed to be caused by the integrated Git instance of VSCode. When I check out and init a repository with my plugin it was wreaking all types of havoc. My fix? I simply turn off the Git pane during init and turn it on afterwords. Problem solved. :)

attie-argentum commented 1 year ago

Very true, though others have reported similar a PermissionError and claim no VSCode... 🙈

Perhaps this is just "locking on Windows", and it cannot be worked around - VSCode isn't the only thing that may have a peek while west is busy after all.

mbolivar-ampere commented 12 months ago

I'm going to close this since it seems like many of the issues here have been diagnosed as environment related problems on Windows that are outside of west's control. If someone is still having an issue that they believe can be attributed to west and which matches the symptom in this issue description ('west init completes but leaves everything in zephyr/manifest-tmp'), feel free to reopen. Otherwise, please open new issues for separate problems to keep this tracker clean.

thewh1teagle commented 7 months ago

I had this issue today too. Looks like vscode indexed things related to git and that's locked the folders while west init tried to delete it. A workround:

  1. I closed vscode completly and opened cmd.exe in that path
  2. Cleaned and ran again 
    rmdir /s .west\
rmdir /s nrf\manifest-tmp\
west init -m https://github.com/nrfconnect/sdk-nrf --mr v2.5.1
=== Initialized. Now run "west update" inside C:\files\projects\side_projects\setup-nrf
marc-hb commented 7 months ago

This can't be "fixed" but it looks frequent enough to deserve a warning and recommended workaround in west init -h. Everyone please review #703 that I just submitted, thanks!

trond-snekvik commented 7 months ago

Although it wouldn't fix this in all scenarios, abandoning the manifest-tmp directory mechanism would resolve this for a lot of users. If the manifest has a self.path, we'd still need to move the directory after cloning (which could result in the same issue), but I'd argue that most manifest repositories don't have this field. Notably, Zephyr itself doesn't.

I understand that manifest-tmp offers a cleaner recovery mechanism on repeated calls to west init, but it also randomly breaks the second step of the getting started procedure on the most popular OS.

It can't be fixed for all scenarios, but it can be fixed for the most common one.

marc-hb commented 7 months ago

I understand that manifest-tmp offers a cleaner recovery mechanism on repeated calls to west init,

There is worse that "repeated calls": consecutive west commands believing that west init was successful when it wasn't. Practically all filesystems guarantee atomicity of renames. This makes rename an extremely useful tool.

BTW the core issue is not just with west and not just with init: git can repack and rename and delete files too. Unsurprisingly, git will do whatever it wants whenever it wants with its files.

Most popular or not, users of a filesystem that does not support concurrency should learn to avoid concurrency. This is good practice and important education that's valid way beyond west and VS code. Maybe the newer ReFS is better? I tried to find information about it but the name is unfortunately unsearchable.

If anything, VS Code and other IDEs should be aware of the limitations of the filesystem they're using and refrain from interfering with random files they don't own. Avoiding background use of files in .git/ and .west/ would be good. They could also pause when they don't have focus; it wouldn't be perfect but it would avoid some failures.