search

DataBiosphere / azul

Metadata indexer and query service used for AnVIL, HCA, LungMAP, and CGP
Apache License 2.0
7 stars 2 forks source link

Job log limit exceeded on GitLab in `prod` #4223

Closed hannes-ucsc closed 2 years ago

hannes-ucsc commented 2 years ago

Log of reindex job https://gitlab.azul.data.humancellatlas.org/ucsc/azul/-/jobs/21605

ends with

2022-05-28 20:53:37,531 INFO    MainThread: Message count history (most recent first) is [34021, 33944, 34231, 34245, 34376, 34259, 34097, 34245, 34515, 34837, 34807, 34777, 35311, 35504, 35162, 35494, 35510, 35160, 35242, 35256, 35449, 35602, 35764, 35587, 35886, 35970, 36453, 36335, 36246, 36898, 36965, 36966, 37606, 37112, 36976, 37014, 37175, 37048, 37211, 37848, 37694, 37795, 37627, 37675, 37786, 37817, 37874, 38431, 38157, 38164, 38332, 38286, 38523, 38319, 38766, 38357, 38672, 39265, 39209, 39399].
2022-05-28 20:53:37,531 INFO    MainThread: Waiting for 4 queue(s) to stabilize ...
2022-05-28 20:53:47,547 DEBUG   MainThread: Queue azul-notificat
Job's log exceeded limit of 8388608 bytes.
Job execution will continue but no more output will be collected.

We need to extend the lob log limit.

achave11-ucsc commented 2 years ago

@hannes-ucsc I found the investigation by Amar on this issue from last time. It seems like the configuration is managed by aws/repositories/azul-gitlab-config/. I might be able to modify the config for dev but not prod (no ssh access), where it matters. How should we proceed?

hannes-ucsc commented 2 years ago

https://us-east-1.console.aws.amazon.com/codesuite/codecommit/repositories/azul-gitlab-config/commit/be5c5a6e6a0b7a700fe259535aa1017112bea563?region=us-east-1

melainalegaspi commented 2 years ago

@hannes-ucsc : "The commit I linked to above fixes this issue."