sreichl
commented
3 months ago - GREAT issues reported by RtH, I could not replicate them
- removing "download_by = tsv" parameter here https://github.com/epigen/enrichment_analysis/blob/d5ffc84178b78b7634799fc49c844956cb491a99/workflow/scripts/region_enrichment_analysis_GREAT.R#L68
- jobs spaced out with more delays that might/could help with random "denial of service attack”-prevention? (dozens/100s of simultaneous jobs seem to trigger failures) → check if I can implement some kind of staggered/sleep approach within Snakemake
- The GREAT server currently accepts a maximum of 5 API requests for all API users at any given time. More than 5 requests will be denied. To make the maximum of 5 API calls you will need to split your jobs and process them separately. A possibly way of processing the data is seen in our [greatBatchQuery.py](https://great-help.atlassian.net/wiki/download/attachments/655447/greatBatchQuery.py?version=1&modificationDate=1345890224000&cacheVersion=1&api=v2) script. Feel free to adapt it to your needs. from https://great-help.atlassian.net/wiki/spaces/GREAT/pages/655447/Programming+Interface
- the most probable cause seems really the many parallel queries. queries are larger when the parameter is =tsv hence it may fail sooner
- RtH: yeah, maybe. But I also reduced the jobs to just 5 and it still had the error
- RtH: Maybe there is some 'ddos' type security for high volume, high payload jobs, but that also doesn't make sense