Collection creation fails intermittently from UI

[gjost]

@sara.beckman Yesterday I tried to create a collection with 192.168.56.101/ui/collection/ddr-densho/new-manual/ I got a 404 time out. The Collection doesn't appear in the Collections list on the Web UI, but it is now in /media/qnfs/kinkura/gold/ddr-densho-386 but with no collection.json just a uuid.log file. I didn't register the ID with the id service

[gjost]

That uuid.log part is especially weird since the DDR code doesn't deal with UUIDs anywhere except in git-annex.

I was able to create new ddr-testing-40151 and ddr-densho-2000 collections with no problems.

Was celery (background worker) running? Can you try again with a ddr-testing-* collection? I'm thinking the timeout may have been significant.

[sarabeckman]

I tried again with a testing collection ddr-testing-40152 and I got the same 404 error. My local VM is running the current master V2.7.1

[sarabeckman]

I checked /media/qnfs/kinkura/gold/ddr-testing-40152 and it also has only a uuid.log file and nothing else.

[pkikawa]

was able to successfully create a new ddr-testing and a new ddr-densho-387 collection.

[gjost]

Is there something different about the two machines you guys are using?

[pkikawa]

no. sara's is a copy of mine. i will give her a fresh copy and have her try again

[sarabeckman]

I got the new copy of the VM, but I'm still getting 404 errors when trying to create a ddr-testing collection. I closed all other Chrome windows and still got a 404.

[pkikawa]

I just tested from another workstation and got a 404. in this case not even the collection directory was created in gold.

[pkikawa]

PYKoldlocal.log PYKlocal.log both were successful. new button first, than new-manual

[sarabeckman]

Here are my log files. The first is my fresh log file and the second is from my longer session today. local.log local3222019.log

Also I tried to use the button to create a new collection for ddr-testing and I also got a 404.

[sarabeckman]

I tried creating a new collection (ddr-densho-389) this morning using 192.168.56.101/ui/collection/ddr-densho/new-manual I got the same 404 page. I then tried creating ddr-testing-40169 with new-manual and also got the same 404 page.

Here is my log file the new-manual section should be at 03-27-2019 from 8:20 am to 8:24 am. Saralocal032719.log

[sarabeckman]

ddr-densho-389 has been created on /media/qnfs/kinkura/gold and like before it only has a uuid.log file.

[sarabeckman]

This morning we (Geoff, Philip and I) changed the local media storage for my local VM on my workstation to point to a local directory /media/ddrtest and I still received the same 404 message when I used the New Collection button to try to create ddr-testing-40170. Appears at 2019-03-28 10:50 Sara032819localmedia.log

Here is also my Celery log - Geoff wanted me to include this too. SaraCelery20190328.log

[sarabeckman]

After testing with the local media drive on my workstation. I used my ddr credentials on Philip's workstation and was able to create a ddr-testing collection.

[sarabeckman]

I was able to successfully create ddr-testing-40176 on using 2.6.8. I used 192.168.56.101/ui/collection/ddr-testing/new-manual and I still got a 404 page, but when I checked the collection at /media/qnfs/kinkura/gold/ddr-testing-40176 it had all the correct files and when I refreshed the collections page in the Editor UI ddr-testing-40176 was listed.

[sarabeckman]

My log file from creating ddr-testing-40176 on 2.6.8 sara2-6-8.log

[sarabeckman]

I was able to successfully create ddr-testing-40177 on using 2.6.9. I used 192.168.56.101/ui/collection/ddr-testing/new-manual and I still got a 404 page, but when I checked the collection at /media/qnfs/kinkura/gold/ddr-testing-40177 it had all the correct files and when I refreshed the collections page in the Editor UI ddr-testing-40177 was listed.

I then tried to create ddr-testing-40178 by clicking on the New Collection button for ddr-testing. I got a 404 page. The collection does not show up in /media/qnfs/kinkura/gold, but it is registered with the ID Service. The collection also does not appear on the Collection List in the UI.

[pkikawa]

Tested on Caitlins laptop. used version 2.7.1 on master. successful collection creation via new-manual AND via webui

[sarabeckman]

Went back to 2.6.8 to try the New Collection button. I then tried to create ddr-testing-40182 by clicking on the New Collection button for ddr-testing. I got a 404 page. The collection does not show up in /media/qnfs/kinkura/gold, but it is registered with the ID Service. The collection also does not appear on the Collection List in the UI. (Same behavior as when I tried New Collection button on 2.6.9).

[pkikawa]

Tested on Akoyet. used version 2.7.1 on master. successful collection creation via new-manual AND via webui

[gjost]

So what patterns do you guys see in all of this? I see credentials as being one possibility.

[pkikawa]

not sure about creds because sara was able to make colelctions on my workstation with her creds and I was able to make collections on Caitlins laptop with her creds. at this point i'm thinking it's maybe a network issue

[sarabeckman]

I successfully created ddr-testing-40187 and ddr-testing-40188 on akoyet 40187 with the button and 40188 with new-manual with my credentials on 2.7.1.

I was unsuccessfully created ddr-testing-40185 and ddr-testing-40186 on pahto with my credentials on 4.7.1. No gold repo was created with 40185 (button). 40186 (new-manual) was created in gold but only with a uuid.log file.

[gjost]

is it possible that the SSH keys used to interact with Gitolite on the hub server are not installed properly? What do you get for this, when logged in as ddr?

ssh git@mits.densho.org info

You should see something like this:

$ ssh git@mits.densho.org info       
hello gjost, this is git@mits2 running gitolite3 3.6.6-1 (Debian) on git 2.11.0 

 R W C  ddr-csujad-[0-9]+                                                       
 R W C  ddr-csujad-[0-9]+-[0-9]+                                                
 R W C  ddr-densho-[0-9]+                                                       
 R W C  ddr-densho-[0-9]+-[0-9]+                                                
 R W C  ddr-dev-[0-9]+                                                         
 R W C  ddr-dev-[0-9]+-[0-9]+                                                  
 R W C  ddr-hmwf-[0-9]+                                                        
 R W C  ddr-hmwf-[0-9]+-[0-9]+                                                 
 R W C  ddr-jamsj-[0-9]+                         
...

If it asks for a password it's not set up properly.

[gjost]

geoff.froh [14:28]

glacier:testcollections Geofff$ git clone git@mits.densho.org:ddr-testing-40186.git
Cloning into ‘ddr-testing-40186’...
remote: Counting objects: 5, done.
remote: Compressing objects: 100% (3/3), done.
remote: Total 5 (delta 0), reused 0 (delta 0)
Receiving objects: 100% (5/5), done.
warning: remote HEAD refers to nonexistent ref, unable to checkout.`

[gjost]

gjost [14:29]

$ ddr clone -u gjost -m gjost@densho.org -i ddr-testing-40186 --dest /var/www/media/ddr/ddr-testing-40186
Traceback (most recent call last):
...
git.exc.GitCommandError: 'git checkout master' returned with exit code 1
stderr: 'error: pathspec 'master' did not match any file(s) known to git.'

[gjost]

geoff.froh [14:39]

root@mits2:/home/git/repositories/ddr-testing-40186.git# git fsck
notice: HEAD points to an unborn branch (master)
Checking object directories: 100% (256/256), done.

[gjost]

sara.beckman [14:44]

Sara (ddrlocal) ddr@DDREditor:/home/densho$ git --version
git version 2.11.0
Sara (ddrlocal) ddr@DDREditor:/home/densho$ git annex version
git-annex version: 6.20170101.1

[gjost]

geoff.froh [15:42] silverstar ddr-testing-40177 on 2.6.9 master (new-manual)

(ddrlocal) ddr@ishigura:/media/qnfs/kinkura/gold/ddr-testing-40177$ ls -la
total 136
drwxr-xr-x   3 ddr ddr   3072 Mar 29 10:07 .
drwxr-xr-x 632 ddr ddr 322560 Mar 29 15:35 ..
-rw-r--r--   1 ddr ddr    394 Mar 29 10:07 changelog
-rw-r--r--   1 ddr ddr   2409 Mar 29 10:07 collection.json
-rw-r--r--   1 ddr ddr    285 Mar 29 09:44 control
-rw-r--r--   1 ddr ddr   2768 Mar 29 10:07 ead.xml
drwxr-xr-x   9 ddr ddr   6144 Mar 29 11:55 .git
-rw-r--r--   1 ddr ddr      9 Mar 29 09:44 .gitignore
(ddrlocal) ddr@ishigura:/media/qnfs/kinkura/gold/ddr-testing-40177$ git status
On branch master
Your branch is based on ‘origin/master’, but the upstream is gone.
 (use “git branch --unset-upstream” to fixup)
nothing to commit, working tree clean

[gjost]

geoff.froh [15:44]

(ddrlocal) ddr@ishigura:~$ git clone git@mits.densho.org:ddr-testing-40177.git
Cloning into ‘ddr-testing-40177’...
warning: You appear to have cloned an empty repository.

[gjost]

I set up temp users and sample repositories on mits2 and another server accessible to the affected machines. I asked @pkikawa to run the following to test Git and git-annex, bypassing Gitolite:

cd /tmp/
git clone USER@REDACTED:~/testing-gitolite.git
cd testing-gitolite
ls -l
# You should see README and several git-annex symlinks.
# Modify README and commit.
# Copy a file into the repo, git-annex-add it and commit.
git annex sync

Both of the tests were successful and I was able to sync with the repositories on my end and see his changes.

[gjost]

I tailed the Gitolite logs while running repository create/clone operations. One run was successful while three runs, on two different machines, failed.

A run on my remote development machine was successful: deb9a-ok.log

This failed run on silverstar in the LAN diverged from the successful run at line 19 silverstart-fail1.log

These failed runs on different machines on the LAN both diverged from the successful run at line 27: ranier-fail.log silverstart-fail2.log

[gjost]

Gitolite logging docs

[gjost]

Notes infodump. 20190329-gitolite.txt

[gjost]

(2019-04-02) We decided to pause this one until we get further info.