Ashley-Tung
commented
4 months ago For Memverge: This is Ticket 3431
Closed yiweizh-memverge closed 4 months ago
Ashley-Tung
commented
4 months ago For Memverge: This is Ticket 3431
Ashley-Tung
commented
4 months ago Adding on section on the same job son 23*:
I also want to add on that Ru would submit the same job on the 23* opcenter on 4/16 and would get a different error. I have attached log bundles to some of the jobs, but it is about 200 jobs as well. She cancelled them because it seemed to hang.
| history.enabled | true | Y | |
| image.cachePath | s3://opcenter-bucket-a7c76ee0-b4a6-11ee-8b4f-0abb2ed0b8bf/images | Y | |
| image.imageOpInterval | 200ms | Y | Y |I believe in this opcenter, the image.cachePath is pointing to a bucket.
The error I keep seeing for all the files is on the stderr and not from opcenter. I have checked their buckets and the files exist with no issue. No folder with the same names exist. I believe she has the 200+ jobs write to the same output file, which could be a concern
Failed to read ../output/Fungen_xQTL.cis_results_db.tsv: [Errno 2] No such file or directory: '../output/Fungen_xQTL.cis_results_db.tsv'[Update from Slack] The issue we were seeing on the 54* opcenter (from this issue: https://github.com/rfeng2023/mmcloud/issues/75) is likely due to the cachePath of the image, which is still pointing to the opcenter
| image.cachePath | file:///mnt/memverge/imagesBecause there were many jobs loading the image, this caused a network traffic issue, causing the jobs to not execute.
TODO: What I suggest for the 54 opcenter, is creating a bucket for the image cache path. I can then upgrade the system variables to point there. After I do that, I can help to upgrade the 54 opcenter to 2.5.4 (just to confirm, this is the old east coast opcenter).
@rfeng2023 please create a bucket with a images and releases folder and let me know the bucket name so I can set it up
For the same jobs that ran on 23 opcenter, Ru confirmed that these are a lower priority job and that we will look at them at the beginning of May when the pipeline can be changed back to accomodate the susie_export runs (so that we may replicate them)
Engineering and I did take a look at the script contents and we have a theory as to why the jobs consistently run into the issue of No such file or directory: '../output/Fungen_xQTL.cis_results_db.tsv'. Looking at the `cis_resultsexport` workflows in fine_mapping_post_processing.ipynb, we can see that step 3 removes files
bash: expand = "${ }", container = container, stderr = f'{_input:n}.stderr', stdout = f'{_input:n}.stdout', entrypoint=entrypoint
rm -rf "${_input:ad}/${name}_cache/"
rm -rf ${_input:an}.temp
And step 2 looks for those files
if (file.exists(paste0(${_output:anr},".temp"))) {
library(tidyverse)Of course, these files don't necessarily correspond to Fungen_xQTL.cis_results_db.tsv itself, AND we do not have the original pipeline script from when this error happened, HOWEVER, we are concerned about this script behavior, because if you have 200+ jobs running the same workflows that will remove/add/write to the SAME file, you are likely to get file issues because of this behavior. Please keep this in mind when we get back to debugging it, but this many jobs writing to the same file could cause unforeseen errors MMC cannot account for.
Please let me know if you have any more questions or concerns.
Ashley-Tung
commented
4 months ago [Update]
Ashley-Tung
commented
4 months ago Closing this issue for now, unless we see it re-appear. Both east coast opcenters should now be on 2.5.5
Ashley-Tung
commented
4 months ago For Memverge: This is ticket 3508
See some sample jobs below:
Failed to complete
https://54.81.85.209/#/opcenter/jobs/8hzjn4rh4euino8yyw45z
fagent.log
From the log we can see that the container exited gracefully with code 0, but the opcenter didn't get the status of it that the it has been done, resulting in Failed to Complete.
Failed to execute
https://54.81.85.209/#/opcenter/jobs/v9o7wky28d58w3obrhmiw
fagent.log
The container has started running, but the opcenter didn't get its status so resulting in Failed to Execute.