dsotirho-ucsc
commented
3 years ago The manifest generation failing with a broad filter (all file types) appears to be caused by exceeding the 15 minute timeout configured for manifest generation.
Unfortunately the logs surrounding the originally reported error are a bit hard to filter through due to multiple manifest requests being made within a short time. With further repeated tests however I was able to locate the logs of when the manifest generation was requested and when it failed and in these cases it was just over 15 minutes when the error occurred. (Note: These tests were for a "dcp1" catalog manifest because the manifest generation succeeded (failed to fail) for the "dcp2" catalog)
1608588399752 2020-12-21T14:06:39.752 dcp1 {"fileFormat":{"is":["bam","csv","csv.gz","docx","fastq","h5","h5ad","loom","mtx.gz","pdf","Robj","tar","tar.gz","tsv","tsv.gz","txt","txt.gz","xlsx","zip"]}}
1608589377781 2020-12-21T14:22:57.781 Caught exception for <function start_manifest_generation_fetch at 0x7f2b4f93c700>
=> 16.30 min
1608591592269 2020-12-21T14:59:52.269 dcp1 {"fileFormat":{"is":["bai","bam","csv","csv.gz","docx","fastq","fastq.gz","matrix","npy","npz","pdf","results","txt","unknown"]}}
1608592516730 2020-12-21T15:15:16.730 Caught exception for <function start_manifest_generation_fetch at 0x7f3e2eebe700>
=> 15.41 min
1608594253495 2020-12-21T15:44:13.495 dcp1 {"fileFormat":{"is":["bai","bam","csv","csv.gz","docx","fastq","fastq.gz","matrix","npy","npz","pdf","results","txt","unknown"]}}
1608595175786 2020-12-21T15:59:35.786 Caught exception for <function start_manifest_generation_fetch at 0x7f3509130700>
=> 15.37 min
1608653538573 2020-12-22T08:12:18.573 dcp1 {"fileFormat":{"is":["bam","csv","csv.gz","docx","fastq","h5","h5ad","loom","mtx.gz","pdf","Robj","tar","tar.gz","tsv","tsv.gz","txt","txt.gz","xlsx","zip"]}}
1608654451137 2020-12-22T08:27:31.137 Caught exception for <function start_manifest_generation_fetch at 0x7fac455ab700>
=> 15.21 min
melainalegaspi
theathorn
amarjandu
nadove-ucsc
hannes-ucsc
The request …
… fails when following the redirect.
CloudWatch Logs Insights
region: us-east-1
log-group-names: /aws/lambda/azul-service-dev-manifest
start-time: 2020-12-18T23:23:10.000Z
end-time: 2020-12-18T23:37:18.000Z
query-string:
Traceback (most recent call last):
File "/var/task/azul/service/manifest_service.py", line 279, in _generate_manifest
base_name = generator.write_to(text_buffer)
File "/var/task/azul/service/manifest_service.py", line 739, in write_to
for hit in self._create_request().scan():
File "/opt/python/elasticsearch_dsl/search.py", line 723, in scan
for hit in scan(
File "/opt/python/elasticsearch/helpers/actions.py", line 443, in scan
resp = client.scroll(
File "/opt/python/elasticsearch/client/utils.py", line 84, in _wrapped
return func(*args, params=params, **kwargs)
File "/opt/python/elasticsearch/client/__init__.py", line 1373, in scroll
return self.transport.perform_request(
File "/opt/python/elasticsearch/transport.py", line 351, in perform_request
status, headers_response, data = connection.perform_request(
File "/opt/python/elasticsearch/connection/http_requests.py", line 161, in perform_request
self._raise_error(response.status_code, raw_data)
File "/opt/python/elasticsearch/connection/base.py", line 229, in _raise_error
raise HTTP_EXCEPTIONS.get(status_code, TransportError)(
elasticsearch.exceptions.NotFoundError: NotFoundError(404, 'search_phase_execution_exception', 'No search context found for id [100290778]')