Cannot export PM results >1k jobs

[koonchaya]

Original post: https://rfcx.slack.com/archives/C03FD1WD02J/p1715001354394219 @carlybatist

Project - https://arbimon.org/project/biosoundscape/analysis/patternmatching

We're experiencing some issues with exporting the PM results for all jobs using the 'Export all results (.csv)' function in Arbimon for the project BioSoundSCapes. This is a critical feature given the volume of PM jobs to export (>1k). The issue that we've run into is in the 'Arbimon export' email from no-reply@arbimon.org, specifically there is no link provided to download the zip file. The 'download' button is not hyperlinked.

@carlybatist could download a much smaller bulk-export job from a separate project ok, so my hunch is that the issue has to do with how many PMs there are in the project (>1k)

[koonchaya]

I tried export all PM from this project. I am able to download the zip file. There are >1k job results in this project but in my export folder contains 333 job results.

occupancy-export (1).zip

[rassokhina-e]

the issue occurs for the job where the PM rois more then 90k results, the main issue related to the time for the query respons. So, I just need to improve the query to get the data for a huge PM job.

[rassokhina-e]

the report is in a progress

[koonchaya]

Export files in https://drive.google.com/drive/folders/1wUXEQeTlEheaNBtJ2ISNXIagu3n2j17M?usp=drive_link

[koonchaya]

Suggested changes to PM bulk export:

• remove all fields up to x1
• reduce any numbers to 3 d.p.
• create a summary file _pattern_matchings.csv

[rassokhina-e]

[koonchaya]

@rassokhina-e

• [x] Change the first filename to _pattern_matching.csv
• [x] Move job id to the end of the filename
• [x] Bigger export cannot be opened see this example file that I tried to export from BCI project on staging

pattern-matching-export (2).zip

[rassokhina-e]

Please recheck on staging

[koonchaya]

[rassokhina-e]

We decreased the PM files, but still have a problem with exporting big PM jobs. The zip folder with all CSV files for the BioSoundSCape project is 2.5 Gb, there is a problem with putting this huge file to aws s3. So we need additional time to fix this problem and implement multipart upload to s3 to split this huge zip to the chunks

[koonchaya]

@rassokhina-e Can we export them into separate files like we do on project backup export, add 200 rows of data in a file? I am not sure it this would help.

[carlybatist]

@koonchaya so just to clarify this was one of the fixes in the new arbimon update right? just saw that it was still open so wanted to check :) thanks!

[koonchaya]

Zhenya just released the another fix yesterday and I was waiting to for the export result first. It seems to work now.