search

MultiQC / MegaQC

Web application to collect and visualise data across multiple MultiQC runs.
http://megaqc.info/
GNU General Public License v3.0
95 stars 27 forks source link

upload error #86

Open vladsavelyev opened 5 years ago

vladsavelyev commented 5 years ago

Hey,

I'm trying to run megaqc on outputs of bcl2fastq module, and getting an error:

megaqc upload multiqc_data.json

This is MegaQC v0.1.dev1

/Users/vsaveliev/miniconda3/envs/multiqc/lib/python3.7/site-packages/megaqc/utils/settings.py:42: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
  configs = yaml.load(f)
 * Environment: test
 * Database type: sqlite
 * Database path: sqlite:////Users/vsaveliev/miniconda3/envs/multiqc/lib/python3.7/site-packages/megaqc.db
[INFO   ]        commands : Uploading file 'multiqc_data.json'
[ERROR  ]          megaqc : Error: JSON response could not be parsed (status code: 400)

Manual debug only showed that the JSON response is in fact not a JSON at all, but an HTML with error 400. Not sure where to go from that.

Attaching the json input (renamed to txt): multiqc_data.txt

Vlad

multimeric commented 5 years ago

Hmm, that's a strange error. I am able to upload the file without issue:

This is MegaQC v0.1.dev1

 * Environment: test
 * Database type: sqlite
 * Database path: sqlite:////home/michael/Programming/MegaQC/megaqc.db
[INFO   ]        commands : Uploading file 'multiqc_data.txt'
[INFO   ]          megaqc : Data upload queued successfully

Since the issue seems to be with the upload, perhaps the version of MultiQC is outdated? Because I believe megaqc upload actually just runs MultiQC behind the scenes.

matthdsm commented 5 years ago

Same issue here, using MultiQC 1.7 and the latest dev version of MegaQC

M

multimeric commented 5 years ago

Are you running megaqc simultaneously (which you need to do)? Can you start it in debug mode so we get a full traceback?

export FLASK_DEBUG=1
megaqc run

Then in another shell:

megaqc upload ...
matthdsm commented 5 years ago

Not much help there :/

(base) [10:25:03] matdsmet:MegaQC git:(master*) $ megaqc run                                                                                        
This is MegaQC v0.1.dev1

 * Environment variable FLASK_DEBUG is true - running in dev mode
 * Environment: dev
 * Debug mode: on
 * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
 * Restarting with stat
This is MegaQC v0.1.dev1

 * Environment variable FLASK_DEBUG is true - running in dev mode
 * Debugger is active!
 * Debugger PIN: 131-391-679
 * Environment: dev
 * Database type: sqlite
 * Database path: sqlite:////Users/matdsmet/miniconda3/lib/python3.7/site-packages/megaqc-0.1.dev1-py3.7.egg/megaqc.db
 * Environment: dev
 * Database type: sqlite
 * Database path: sqlite:////Users/matdsmet/miniconda3/lib/python3.7/site-packages/megaqc-0.1.dev1-py3.7.egg/megaqc.db

127.0.0.1 - - [20/Aug/2019 10:25:40] "POST / HTTP/1.1" 200 -
127.0.0.1 - - [20/Aug/2019 10:26:24] "POST / HTTP/1.1" 200 -
(base) [10:26:14] matdsmet:MegaQC git:(master*) $ megaqc upload ../multiqc_data.json
This is MegaQC v0.1.dev1

 * Environment: test
 * Database type: sqlite
 * Database path: sqlite:////Users/matdsmet/miniconda3/lib/python3.7/site-packages/megaqc-0.1.dev1-py3.7.egg/megaqc.db
[INFO   ]        commands : Uploading file '../multiqc_data.json'
[ERROR  ]          megaqc : Error: JSON response could not be parsed (status code: 200)
multimeric commented 5 years ago

Hmm so it possibly is succeeding. It's just returning HTML as noted above. I'll have a look into why tomorrow. Can you check if it's visible in the samples page after upload?

ewels commented 5 years ago

Hi all,

Sorry for the slow response. I've seen this before when the MegaQC configuration for MultiQC is wrong and not set to the correct upload API endpoint. Could you please paste the contents of your multiqc.config files? (It should be copied and pasted from the MegaQC config page).

Phil

matthdsm commented 5 years ago

Hi @TMiguelT, @ewels

The samples are not visible in the interface, even though the return has a 200 status. Multiqc config files contain just the megaqc_url needed for the upload command.

> cat ~/.multiqc_config.yaml
megaqc_url: http://localhost:5000

Anything I'm missing? What do you mean by the MegaQC config page? https://multiqc.info/megaqc page seems to non existent.

Thanks M

multimeric commented 5 years ago

I believe he's referring to the config page within MegaQC, e.g. http://localhost:3000/users/multiqc_config. You need to have config that looks a bit like this:

megaqc_url: http://localhost:3000/api/upload_data
megaqc_access_token: 0UAbE4kR3PkAJHHQeBFXtosQh0k0i9WM3f7y7W92yZ4msR0NafoJCr8LoQVnTOYEHlrhJpFKMjysibl2
ewels commented 5 years ago

That’s what I thought - it’s returning a 200 code because you are successfully loading the homepage for MegaQC. We should probably try to be more clever with HTTP headers and user agents and stuff to check for this edge case.

See https://github.com/ewels/MegaQC/blob/master/docs/usage-setup.md#multiqc-configuration for instructions. In your running MegaQC installation you should have a navigation drop down page with config details for MultiQC. The URL is not the MegaQC homepage.

matthdsm commented 5 years ago

Makes sense! Sorry for missing the instructions. Seems to work a-ok now!

Thanks! M

multimeric commented 5 years ago

Oh, I see the problem. As Phil said, it's submitting to the wrong URL (the homepage), but because MegaQC ignores the POST parameters it just treats it as a normal GET request and returns the HTML with the 200 status code. So not only should that endpoint not work with a POST, but it needs to fail if the Accept header isn't Accept: text/html. There might also be some other checks we need in place that I haven't thought of.