User doesn't get all PM export from biosoundscape project

rfcx / arbimon-legacy

https://arbimon.org

Apache License 2.0

0 stars 0 forks source link

User doesn't get all PM export from biosoundscape project #1561

Closed koonchaya closed 2 months ago

koonchaya commented 3 months ago

Original report: https://rfcx.slack.com/archives/C03FD1WD02J/p1717598848034249

Our favorite PM-bulk-export project is at it again (lol). This project ran another PM bulk export 3 days ago and haven't gotten anything. I know we had errors with this project before but had fixed them - Can you confirm that this export is indeed running?https://arbimon.org/p/biosoundscape

rassokhina-e commented 3 months ago

We increased the request memory to 2500Mi, the job went to 2000Mi and was terminated, but there was 500Mi free.

Also, we improved the --max-old-space-size for the export jobs.

So, it just takes to rerun a job and we can do it manually Untitled 2 (2) Untitled 2 (1)

carlybatist commented 3 months ago

@rassokhina-e so you want me to have the user re-request the bulk export then?

rassokhina-e commented 3 months ago

No, thanks! We manually exported the report

-signed url https://arbimon.s3.eu-west-1.amazonaws.com/exports/3941/2024-06-06%2011%3A02%3A04/pattern-matching-export.zip?AWSAccessKeyId=AKIA45B7YWBX4KWANUBB&Expires=1718289381&Signature=0laBEZtg4N0fcI2SoDZIDn3A9Cs%3D

rassokhina-e commented 3 months ago

we sent report to the user on the prev week. The current state of the task still in a progress, because I'm checking the memory leak during exporting big PM reports now

rassokhina-e commented 3 months ago

I tested the huge PM export for the biosoundscape project and checked the memory leak (attached image). The memory usage was reduced by refactored code, so we just need to retest it on the real project after the production release

rassokhina-e commented 3 months ago

@koonchaya please check the PM export on the staging side

koonchaya commented 3 months ago

@rassokhina-e I haven't got the PM export that I requested from staging.

rassokhina-e commented 3 months ago

@koonchaya Fixed

rassokhina-e commented 3 months ago

Hi @carlybatist ! Can you please run export al PM jobs on the biosoundscape project?

koonchaya commented 3 months ago

@carlybatist @rassokhina-e I will do it.

koonchaya commented 3 months ago

I got the export in the mail but the export size is 2.4gb. I am trying to download it but it will take a long time.

koonchaya commented 3 months ago

Exported files match with jobs