error in uploading new data

[mfazel]

Hi,

I uploaded a sumstat file a few days ago and there was no error. Since yesterday I've been uploading another file which I don't think is any different that the other one (in terms of columns and structure) and it does not finish completely. It gets uploaded but after a minute or two I see a red error next to the study name and also logs does not show any error. First, I thought it was sorting, then I thought chromosome naming, then I thought alleles, then.... I uploaded it many times and nothing but failed. Then I downloaded the file I had uploaded last week and uploaded it again and see the same error. So is anything wrong with the upload process on your server or anything with my account possibly? Any help will be appreciated.

Thanks, Mehdi

[abought]

Thanks for the note. Indeed, I'd like to dig into this problem further.

We haven't changed the upload process (though a new release is pending that makes chromosome names stricter). Can you send me a link or ID to one of the failed uploads? (assuming it is ok for me to look at the contents, for debugging puposes only)

One edge case issue we have been seeing of late: some users were updating malformed files in a way that triggered massive server memory usage, and as a result we were forced to terminate several upload jobs outside of the normal error logging procedure.

This should only have affected a few problem files, but I'll try to identify the cause of upload failures at the same time as offering actionable help to get your plots working again.

[abought]

Also, we never want our users to feel that they have to resort to trial and error to get an upload working. If you have suggestions for user feedback or docs, let me know!

Here is some advice on recommended file contents. It's a bit broad because we try to support many upload formats, but it covers the basic information expected. https://my.locuszoom.org/about/#prepare-data

[mfazel]

Here is the the log file: [ingest][2022-05-23T02:45:13+00:00] Performing upload step: Calculate SHA256 [success][2022-05-23T02:45:13+00:00] Step completed [success] The GWAS file passed validation. Read the logs carefully, in case any specific lines failed to parse. [ingest][2022-05-23T02:45:13+00:00] Performing upload step: Normalize GWAS file format [success][2022-05-23T02:45:44+00:00] Step completed [ingest][2022-05-23T02:45:44+00:00] Performing upload step: QQ plots and top hit detection [success][2022-05-23T02:46:05+00:00] Step completed [ingest][2022-05-23T02:46:05+00:00] Performing upload step: Prepare a manhattan plot [success][2022-05-23T02:46:22+00:00] Step completed

ID for the summary stat I uploaded today and got the error: 810731 I used the same file last week and it worked. One major suggestion I have is to provide small example files (from each type or format) for users to look at the content or use for test. I did not find one in the upload section where users start the process of uploading. I also looked at github but don't believe there was one. It would help to use as a test in these situations. I even downloaded the same summary statistic from last week and uploaded again but at the end, got the red error next to study name so wasn't sure if it was my file, my computer connection or something else. (the text file size is 37M)

Thanks.

[mfazel]

Hi Andy, Today I tried to upload the same file and it was successful assuming the issue whatever it was is resolved. Thanks

[abought]

Thanks for the followup!

Several users reported a similar issue on the mailing list. The root cause was intermittent email server issues over the weekend (the job was marked as failed if any step- even the final email- has an error. We may revisit that particular edge case in the future)

Things appear to be working normally now, but please let me know if problems recur.