ted-strauss-K1
commented
5 years ago Is there another way to force the run of post-update hook, in this circumstance?
Open yarikoptic opened 5 years ago
ted-strauss-K1
commented
5 years ago Is there another way to force the run of post-update hook, in this circumstance?
yarikoptic
commented
5 years ago The only workaround is to login to the remote host and run the hook manually. Did you hit the same problem I guess @tedstrauss ?
ted-strauss-K1
commented
5 years ago Ya we're testing something out that extracts no metadata, and the web UI shows this:
ERROR: Could not find any metadata directory. Sorry. Most probably cause is that 'publish' didn't run the post-update hook
yarikoptic
commented
5 years ago hm, I received notification of the post , but no comment is any longer posted here.
So -- if you ran ./post-update from under the hooks and "it didn't help" - check out the logs it created in that repo in files .git/datalad/logs/datalad-publish-hook-201* - they might tell what went wrong
paiva
commented
5 years ago Hi, I took down the post because when I checked the logs I found out that my git-annex version was not updated. Upgrading it fixed the issue. Thanks!
paiva
commented
5 years ago Actually, it solved part of the issue, the new error says
[ERROR] 'Tree' object has no attribute '_name' [base.py:iter_items:1193] (AttributeError)
E: no datalad found - skipping generation of indexes for web frontend
fatal: Needed a single revisionBasically, I am unable to run the datalad ls -a --json file . command. Any ideas as to why this could be happening? The frontend looks like this:
It could also be related to this: https://github.com/rtfd/readthedocs.org/issues/4371, maybe I am missing something related to the git submodule?
Thank you so much
yarikoptic
commented
5 years ago could you share output of git status in that repo? is it on a branch and if everything "kosher"
yarikoptic
commented
5 years ago indeed might relate to submodules - please verify that you have them consistent as well (all listed in .gitmodules are installed etc)
paiva
commented
5 years ago Hi! Thanks for your quick reply. Highly appreciated it!
Here is the output for git status and git branch:
I don't know why are all these files deleted, I extracted metadata from these files on another server.
paiva
commented
5 years ago Here is the output of .gitmodules:
root@datalad:/var/www/html# cat .gitmodules
[submodule "397552"]
path = 397552
url = ./397552
[submodule "421863"]
path = 421863
url = ./421863
[submodule "435668"]
path = 435668
url = ./435668
[submodule "623541"]
path = 623541
url = ./623541
[submodule "698527"]
path = 698527
url = ./698527
[submodule "833826"]
path = 833826
url = ./833826
[submodule "963271"]
path = 963271
url = ./963271
[submodule "964522"]
path = 964522
url = ./964522
[submodule "992086"]
path = 992086
url = ./992086
[submodule "999876"]
path = 999876
url = ./999876datalad ls -r .
yarikoptic
commented
5 years ago Disregard above request. 964522 is listed as a submodule but also was committed as a directory?
paiva
commented
5 years ago 964522 is a subdataset that I added in the end. I used the following commands:
datalad create -f -d . <subdir>/
cd <subdir>/
git config -f .datalad/config --add datalad.metadata.nativetype <nifti1 or minc>
datalad aggregate-metadata --force-extraction -d . -r --update-mode all
datalad -f json_pp metadata --recursive --reporton datasets
datalad save -r -d . --message "some commit message" --all-updatedIt should be a submodule and not a directory. Should I run datalad ls -r?
Thanks again!
yarikoptic
commented
5 years ago those commands look good (no need for datalad ls -r atm) . my point is that I think that the super dataset managed to get <subdir>/ added as a subdirectory at some point somehow. git log <subdir> should tell you when. Might it be a manifestation of https://github.com/datalad/datalad/issues/3139 ?
ted-strauss-K1
commented
5 years ago @yarikoptic just to clarify: The web UI should display the folders and treat them as submodules if:
data create -d . <subdir>cd <subdir>; datalad save -r -d .cd ..; datalad save -r -d .datalad publish -r --to REMOTEIs that right?
yarikoptic
commented
5 years ago That all sounds right. I would advise to give (commit) messages to save in the future.
But let me repeat the question: git log <subdir> should (well - might) tell what command added directly to super dataset. If, given that list of commands, you could reproduce it - even better. If not, may be running history in that shell would give a clue how you diverged
paiva
commented
5 years ago yarikoptic
commented
5 years ago Here is the minimalistic script to reproduce the situation (from our online debugging session):
datalad create /tmp/superds
cd /tmp/superds
mkdir subds
touch subds/file
datalad add subds/file
cd subds
datalad create -f . # That was the point of "crime" -- creating
touch file2
cd ..
datalad add subds # worked correctly -- added within subds
git status # super still thinks things belong to it (as well)
datalad save -m "saving recursively upstairs" -r . # saves files also to superdataset
git log --statso possible ideas are
-f|--force create must refuse if the path (directory) contains any file under git control "upstairs", i.e. the check could be as simple as
git -C .. log $PWD | grep -q . && echo not-kosher || echo all-goodWe are implementing "prevention" mechanism for such situations to happen: https://github.com/datalad/datalad/pull/3211
yarikoptic
commented
5 years ago @paiva could you please update us either there was success in your endeavor? ;-)
If publish only uploaded sub-datasets (with data or not) we need to run post-update hook (if web UI is installed) so it could update "web metadata" to reflect sizes etc
related was (and fixed): https://github.com/datalad/datalad/issues/1658 - to run the hook if just publishing the data