Saving the dataset state fails with `permission denied [err: 'add: 1 failed']`

vncntprvst commented 1 year ago

Is it a new bug?

I did not find an existing issue about my problem

What is the problem?

When saving the contents of a dataset (Dataset commands → Save the state in a dataset), the operation fails most of a time after a few %. The log error shows as: git-annex: path/to/dataset/myfile: hGetBuf: permission denied (Permission denied) [err: 'add: 1 failed']

What steps will reproduce the problem?

This is on Windows 10 Enterprise. The dataset repository is a GIN sibling, created following the docs' procedure.

Reproduce the bug:

Open the GUI.
Right click on the dataset, select Dataset commands → Save the state in a dataset`

Additional context

No response

How did you install the App?

[X] Via the Windows installer
[ ] Using pip (pip install datalad-gooey)
[ ] Using brew (brew install datalad-gooey)
[ ] Using conda (conda install -c forge datalad-gooey)
[ ] Not sure
[ ] Other

System information

Diagnostic infos

``` # WTF ## configuration ## credentials - keyring: - active_backends: - Windows WinVaultKeyring - Windows RegistryKeyring - EncryptedKeyring with not defined v.1.0 at C:\Users\wanglab\AppData\Local\Python Keyring\wincrypto_pass.cfg - PlaintextKeyring with no encyption v.1.0 at C:\Users\wanglab\AppData\Local\Python Keyring\keyring_pass.cfg - config_file: C:\Users\wanglab\AppData\Local\Python Keyring\keyringrc.cfg - data_root: C:\Users\wanglab\AppData\Local\Python Keyring ## datalad - version: 0.17.6 ## dependencies - annexremote: 1.6.0 - boto: 2.49.0 - cmd:annex: 10.20220927-ga92546587 - cmd:bundled-git: UNKNOWN - cmd:git: 2.37.3.windows.1 - cmd:ssh: 8.1p1 - cmd:system-git: 2.37.3.windows.1 - cmd:system-ssh: 8.1p1 - humanize: 4.4.0 - iso8601: 1.0.2 - keyring: 23.9.3 - keyrings.alt: 4.2.0 - msgpack: 1.0.4 - platformdirs: 2.5.2 - requests: 2.28.1 - tqdm: 4.64.1 ## environment - GIT_TERMINAL_PROMPT: 0 - PATH: C:\Users\wanglab\AppData\Local\datalad.org\datalad-gooey\python39\Lib\site-packages\PySide6;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.0\bin;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.0\libnvvp;V:\Program Files (x86)\VMware\VMware Workstation\bin\;C:\Program Files\AdoptOpenJDK\jdk-11.0.8.10-hotspot\bin;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.0\bin;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.0\libnvvp;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v9.1\bin;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v9.1\libnvvp;V:\ProgramData\Anaconda3;V:\ProgramData\Anaconda3\Library\mingw-w64\bin;V:\ProgramData\Anaconda3\Library\usr\bin;V:\ProgramData\Anaconda3\Library\bin;V:\ProgramData\Anaconda3\Scripts;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\OpenSSH\;V:\Program Files\MATLAB\R2021b\bin;V:\Program Files\MATLAB\R2020b\runtime\win64;V:\Program Files\MATLAB\R2020b\bin;C:\Program Files\Microsoft SQL Server\120\Tools\Binn\;C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common;V:\Program Files\MATLAB\R2018b\bin;C:\Program Files (x86)\Graphviz2.38\bin;C:\Program Files (x86)\ffmpeg-4.2.1\bin;C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\bin;C:\Program Files\MySQL\MySQL Server 8.0\bin;C:\Strawberry\c\bin;C:\Strawberry\perl\site\bin;C:\Strawberry\perl\bin;C:\Program Files\Microsoft MPI\Bin;V:\Program Files\MATLAB\R2019a\bin\win64;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\WINDOWS\System32\OpenSSH\;C:\Program Files\NVIDIA Corporation\Nsight Compute 2020.1.1\;C:\Users\wanglab\AppData\Local\Bonsai;C:\Program Files\WhiskerTracking\bin;C:\Program Files\Git\cmd;C:\Program Files\Docker\Docker\resources\bin;C:\ProgramData\DockerDesktop\version-bin;C:\Program Files\MySQL\MySQL Shell 8.0\bin\;C:\Users\wanglab\AppData\Local\Microsoft\WindowsApps;V:\Program Files\JetBrains\PyCharm Professional Edition with Anaconda plugin 2019.1.1\bin;C:\Users\wanglab\AppData\Local\Programs\Microsoft VS Code\bin;C:\Users\wanglab\AppData\Local\GitHubDesktop\bin;C:\Program Files\JetBrains\PyCharm Community Edition 2020.2\bin;;C:\Users\wanglab\AppData\Local\Microsoft\WindowsApps;;C:\Users\wanglab\AppData\Local\Programs\Microsoft VS Code Insiders\bin ## extensions - gooey: - description: Gooey (GUI) - entrypoints: - datalad_gooey.askpass.GooeyAskPass: - class: GooeyAskPass - load_error: None - module: datalad_gooey.askpass - names: - gooey-askpass - gooey_askpass - datalad_gooey.gooey.Gooey: - class: Gooey - load_error: None - module: datalad_gooey.gooey - names: - datalad_gooey.lsdir.GooeyLsDir: - class: GooeyLsDir - load_error: None - module: datalad_gooey.lsdir - names: - gooey-lsdir - gooey_lsdir - datalad_gooey.status_light.GooeyStatusLight: - class: GooeyStatusLight - load_error: None - module: datalad_gooey.status_light - names: - gooey-status-light - gooey_status_light - load_error: None - module: datalad_gooey - version: 0.1.0+9.g2f5814c - next: - description: What is next in DataLad - entrypoints: - datalad_next.create_sibling_webdav.CreateSiblingWebDAV: - class: CreateSiblingWebDAV - load_error: None - module: datalad_next.create_sibling_webdav - names: - create-sibling-webdav - datalad_next.credentials.Credentials: - class: Credentials - load_error: None - module: datalad_next.credentials - names: - datalad_next.tree.TreeCommand: - class: TreeCommand - load_error: None - module: datalad_next.tree - names: - tree - load_error: None - module: datalad_next - version: 0.6.1 ## git-annex - build flags: - Assistant - Webapp - Pairing - TorrentParser - MagicMime - Benchmark - Feeds - Testsuite - S3 - WebDAV - dependency versions: - aws-0.22 - bloomfilter-2.0.1.0 - cryptonite-0.29 - DAV-1.3.4 - feed-1.3.2.0 - ghc-8.10.7 - http-client-0.7.9 - persistent-sqlite-2.13.0.3 - torrent-10000.1.1 - uuid-1.3.15 - yesod-1.6.1.2 - key/value backends: - SHA256E - SHA256 - SHA512E - SHA512 - SHA224E - SHA224 - SHA384E - SHA384 - SHA3_256E - SHA3_256 - SHA3_512E - SHA3_512 - SHA3_224E - SHA3_224 - SHA3_384E - SHA3_384 - SKEIN256E - SKEIN256 - SKEIN512E - SKEIN512 - BLAKE2B256E - BLAKE2B256 - BLAKE2B512E - BLAKE2B512 - BLAKE2B160E - BLAKE2B160 - BLAKE2B224E - BLAKE2B224 - BLAKE2B384E - BLAKE2B384 - BLAKE2BP512E - BLAKE2BP512 - BLAKE2S256E - BLAKE2S256 - BLAKE2S160E - BLAKE2S160 - BLAKE2S224E - BLAKE2S224 - BLAKE2SP256E - BLAKE2SP256 - BLAKE2SP224E - BLAKE2SP224 - SHA1E - SHA1 - MD5E - MD5 - WORM - URL - X* - operating system: mingw32 x86_64 - remote types: - git - gcrypt - p2p - S3 - bup - directory - rsync - web - bittorrent - webdav - adb - tahoe - glacier - ddar - git-lfs - httpalso - borg - hook - external - supported repository versions: - 8 - 9 - 10 - upgrade supported from repository versions: - 2 - 3 - 4 - 5 - 6 - 7 - 8 - 9 - 10 - version: 10.20220927-ga92546587 ## location - path: D:\Vincent - type: directory ## metadata_extractors - annex: - distribution: datalad 0.17.6 - load_error: None - module: datalad.metadata.extractors.annex - audio: - distribution: datalad 0.17.6 - load_error: ModuleNotFoundError(No module named 'mutagen') - module: datalad.metadata.extractors.audio - datacite: - distribution: datalad 0.17.6 - load_error: None - module: datalad.metadata.extractors.datacite - datalad_core: - distribution: datalad 0.17.6 - load_error: None - module: datalad.metadata.extractors.datalad_core - datalad_rfc822: - distribution: datalad 0.17.6 - load_error: None - module: datalad.metadata.extractors.datalad_rfc822 - exif: - distribution: datalad 0.17.6 - load_error: ModuleNotFoundError(No module named 'exifread') - module: datalad.metadata.extractors.exif - frictionless_datapackage: - distribution: datalad 0.17.6 - load_error: None - module: datalad.metadata.extractors.frictionless_datapackage - image: - distribution: datalad 0.17.6 - load_error: ModuleNotFoundError(No module named 'PIL') - module: datalad.metadata.extractors.image - xmp: - distribution: datalad 0.17.6 - load_error: ModuleNotFoundError(No module named 'libxmp') - module: datalad.metadata.extractors.xmp ## metadata_indexers ## python - implementation: CPython - version: 3.9.13 ## system - distribution: 10/10.0.19044/SP0/Multiprocessor Free - encoding: - default: utf-8 - filesystem: utf-8 - locale.prefered: cp1252 - filesystem: - CWD: - path: D:\Vincent - HOME: - path: C:\Users\wanglab - TMP: - path: C:\Users\wanglab\AppData\Local\Temp - max_path_length: 259 - name: Windows - release: 10 - type: nt - version: 10.0.19044```

welcome[bot] commented 1 year ago

Welcome Banner (Image: CC-BY license, The Turing Way Community, & Scriberia. Zenodo. http://doi.org/10.5281/zenodo.3332808) Hi there, and thank you for filing an issue. We're excited to have your input and welcome your idea! :blush:

adswa commented 1 year ago

Hi, thanks for the issue! I'll try to figure out what might be wrong, but I need a bit more information. Do I read it correctly that you're seeing this error at the step "create a dataset" from the first part of the docs, i.e., in a local DataLad dataset you created? If so, did you see anything unexpected when you ran datalad create? And is this with any file right from the start, or has it suddenly started to happen with a specific file? Or is it with a Dataset that you cloned from Gin to your local system, following the step "Retrieve data" from the last part of the docs? Or in an entirely different step?

vncntprvst commented 1 year ago

Hi Adina, Thanks for your reply. This issue occurs within an existing dataset. I created it previously, and pushed updates to the GIN repo on gin.g-node.org. I want to commit new data to the repository, so I use the command Save the state in a dataset before pushing. The permission denied occurs almost every time, but not necessarily for the same file.

Also, this may be unrelated but just to mention it, I don't seem to have set up the connection to the repo properly in DataLad Gooey. I get this error when pushing data to the remote repo with Dataset commands → Push data/updates to a sibling

CommandError: '"git" "-c" "diff.ignoreSubmodules=none" "fetch" "origin" "git-annex"' failed with exitcode 128 [err: 'datalad sshrun: line 1: datalad: command not found
fatal: Could not read from remote repository.

Please make sure you have the correct access rights

Instead I commit and push data with git, via a terminal, which works fine.

bpoldrack commented 1 year ago

@vncntprvst

Instead I commit and push data with git, via a terminal, which works fine.

Does datalad push via terminal also work fine? Because the error message you get via gooey seems strange (datalad: command not found), suggesting there's something wrong with the environment/installation.

mih commented 1 year ago

@christian-monch can you please have a look here? I think this is a side-effect of the encapsulated installation of the Python env that contains the datalad installation used for the GUI.

The question to me is whether we should consider this windows specific, or rather a general issue of an environment-based installation. We can only assume the CLI entrypoints to exist within that env, hence all shell commands must run within that env too. However, (currently) it is not guaranteed that a CLI setup exists for any environment that can run the GUI.

This is (likely) also causing failure of the following components:

asking a password from Git/SSH (datalad-gooey-askpass)
datalad special remote implementations (datalad, archives)
XLDRA git-annex backend, and git-remote-datalad-annex

adswa commented 1 year ago

Ok, so then that's two issue - one is the yet unclear permission error, and then there is the environment.

Does datalad push via terminal also work fine?

If you only have datalad-gooey installed, no other DataLad installation, I would expect that you wouldn't be able to call datalad push from a terminal, @vncntprvst. I'd expect something just like datalad: command not found. The installation is deliberately encapsulated in %LOCALAPPDATA%, and the installer doesn't add DataLad's binaries to the env/PATH. But the error message datalad: command not found suggests that some processes fail to find it, and that we might need to add it to PATH. Tagging @christian-monch

vncntprvst commented 1 year ago

I confirm that DataLad cannot be called from a terminal and so datalad push fails. Would you recommend that I add it to the path?

adswa commented 1 year ago

I confirm that DataLad cannot be called from a terminal and so datalad push fails. Would you recommend that I add it to the path?

I'm unsure whether it might not simply be easier to have us fix these early gooey issues and have the next installer take care of doing things more properly. If you are using the command line anyway and have Python already on your system, an installation with a package manager of your choice (conda, pip) would give you the same or more flexibility - If you don't use Python on your system or aren't very keen on using the command line and need things to work before the next installer has taken care of the issue, the binaries are located in %LOCALAPPDATA%\datalad.org\datalad-gooey. On my Windows 10 system this translates to C:\Users\adina\AppData\Local\datalad.org\datalad-gooey - in there, it is part of the packages in python39.

But other than the push, I think the permission error is also quite an interesting one. Were you able to run datalad status or git status in the dataset? Is the dataset only used by you, or are there several users with access to it?

vncntprvst commented 1 year ago

Sounds good. I'm the only one to use this dataset. From a terminal, git status works, not datalad, as it is not in the path. Just one last note, this might be expected, but if I run: %LOCALAPPDATA%\datalad.org\datalad-gooey\python39\python.exe -m datalad status I get this error: The DataLad AutomagicIO functionality has been moved to an extension package. Please install the Python package ``datalad_deprecated`` to be able to use it.

adswa commented 1 year ago

My apologies for the slow ping pong in this issue. Next Tuesday is a an official software demo of this, and most energy currently goes into finishing the metadata functionality until then. If you happen to have time today at 5PM CE(S)T (I think thats 11AM your time), would you mind joining our office hour to take a closer look via screen sharing?

christian-monch commented 1 year ago

PRs #364 and #370 fix the described issue with saving for me. They also allow to create a gin sibling and to push to it.

An issue that is left is that ssh will interact with the terminal in order to as for trust into a new host key. This is not visible on the UI and might give the impression that datalad_gooey is frozen, while it is only waiting on console input (described here issue #371)

With both PRs merged I was able to create a GIN sibling and push to it via the GUI.

vncntprvst commented 1 year ago

Thanks for working on these fixes! @adswa, sorry I missed your message. My apologies. I'll get back to you as soon as I got a chance to test this out.

datalad / datalad-gooey