Can't publish a file set

[jeffhhk]

I was able to upload 0.5GB node tsv and a 2.1GB edge tsv. Those eventually finished, but afterward, clicking the "Done Uploading" button failed.

The HTTP request as curl, less the session cookie:

curl 'https://archive.translator.ncats.io/archive/publish/yeastsrireferencekgtsv/1.0' -H 'User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:90.0) Gecko/20100101 Firefox/90.0' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8' -H 'Accept-Language: en-US,en;q=0.5' --compressed -H 'Connection: keep-alive' -H 'Referer: https://archive.translator.ncats.io/upload?kg_id=yeastsrireferencekgtsv&kg_name=yeast-sri-reference-kg-tsv&kg_version=1.0&submitter_name=Jeff+Henrikson' -H 'Cookie: AIOHTTP_SESSION=...' -H 'Upgrade-Insecure-Requests: 1' -H 'Sec-Fetch-Dest: document' -H 'Sec-Fetch-Mode: navigate' -H 'Sec-Fetch-Site: same-origin' -H 'Sec-Fetch-User: ?1'

HTTP response headers:

HTTP/1.1 504 Gateway Time-out
Server: nginx/1.18.0 (Ubuntu)
Date: Thu, 22 Jul 2021 23:00:02 GMT
Content-Type: text/html
Content-Length: 176
Connection: keep-alive

[RichardBruskiewich]

Hi @jeffhhk thanks for this bug report.

I was worried that the new archive was going to have some hidden bugs. We'll take a look at this ASAP.

[RichardBruskiewich]

Hi @jeffhhk, I think the first part of the mystery is solved here: yes... Gateway inaccessible because the docker container running the archive crashed 4 days ago!

One has to suspect some kind of Docker (memory allocation) resource limitation. It's not every day that one attempts to move gigabytes of data around!

Let me probe a bit deeper...

[RichardBruskiewich]

@jeffhhk, I think your data made it into the repository the first time, except that perhaps you didn't upload a KGX meta-knowledge-graph JSON file (the so called "content metadata"). Did you generate one and attempt to upload it? No worries (for now) if you didn't. We can try to generate and upload one later.

I've attempted to test the KGX validation - which I've attempted to run a background co-routine task - but I'm lead to expect that current design doesn't properly work. I've disabled the validation for now and I've put a disclaimer about KGX validation of files on the 'upload' page, pending a resolution of the issue.

We are continuing our iterations on the code on our side and will update the production server as we become confident of the new or revised code. For now, we hope that the site is basically functional for uploads and downloads of some of the files currently available within the community.

[RichardBruskiewich]

@jeffhhk, I'm going to close this issue for now, but feel free to reopen it should the need arise.

[jeffhhk]

Fix confirmed.