handle Jobs lifespan

[epifanio]

After a while, a succesfully executed jobis no longer available.

Asking for an "expired" job will return a 500 error code on the client side (Internal server error)

the following is the traceback log server side:

pygeoapi_pygeoapi.1.to5npjm7wwdq@server.dns    | RESULTS status PENDING
pygeoapi_pygeoapi.1.to5npjm7wwdq@server.dns    | [2023-01-27T02:12:26Z] {/usr/lib/python3/dist-packages/flask/app.py:1454} ERROR - Exception on /jobs/b506cb48-9d8a-11ed-af5b-02420a000a06 [GET]
pygeoapi_pygeoapi.1.to5npjm7wwdq@server.dns    | Traceback (most recent call last):
pygeoapi_pygeoapi.1.to5npjm7wwdq@server.dns    |   File "/usr/lib/python3/dist-packages/flask/app.py", line 2070, in wsgi_app
pygeoapi_pygeoapi.1.to5npjm7wwdq@server.dns    |     response = self.full_dispatch_request()
pygeoapi_pygeoapi.1.to5npjm7wwdq@server.dns    |   File "/usr/lib/python3/dist-packages/flask/app.py", line 1515, in full_dispatch_request
pygeoapi_pygeoapi.1.to5npjm7wwdq@server.dns    |     rv = self.handle_user_exception(e)
pygeoapi_pygeoapi.1.to5npjm7wwdq@server.dns    |   File "/usr/lib/python3/dist-packages/flask/app.py", line 1513, in full_dispatch_request
pygeoapi_pygeoapi.1.to5npjm7wwdq@server.dns    |     rv = self.dispatch_request()
pygeoapi_pygeoapi.1.to5npjm7wwdq@server.dns    |   File "/usr/lib/python3/dist-packages/flask/app.py", line 1499, in dispatch_request
pygeoapi_pygeoapi.1.to5npjm7wwdq@server.dns    |     return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
pygeoapi_pygeoapi.1.to5npjm7wwdq@server.dns    |   File "/pygeoapi/pygeoapi/flask_app.py", line 361, in get_jobs
pygeoapi_pygeoapi.1.to5npjm7wwdq@server.dns    |     return get_response(api_.get_jobs(request, job_id))
pygeoapi_pygeoapi.1.to5npjm7wwdq@server.dns    |   File "/pygeoapi/pygeoapi/api.py", line 195, in inner
pygeoapi_pygeoapi.1.to5npjm7wwdq@server.dns    |     headers, status, content = func(*args, **kwargs)
pygeoapi_pygeoapi.1.to5npjm7wwdq@server.dns    |   File "/pygeoapi/pygeoapi/api.py", line 177, in inner
pygeoapi_pygeoapi.1.to5npjm7wwdq@server.dns    |     return func(cls, req_out, *args[2:])
pygeoapi_pygeoapi.1.to5npjm7wwdq@server.dns    |   File "/pygeoapi/pygeoapi/api.py", line 3244, in get_jobs
pygeoapi_pygeoapi.1.to5npjm7wwdq@server.dns    |     if JobStatus[job_['status']] in (
pygeoapi_pygeoapi.1.to5npjm7wwdq@server.dns    |   File "/usr/lib/python3.10/enum.py", line 440, in __getitem__
pygeoapi_pygeoapi.1.to5npjm7wwdq@server.dns    |     return cls._member_map_[name]
pygeoapi_pygeoapi.1.to5npjm7wwdq@server.dns    | [ERROR: 2023-01-27 02:12:26 : pygeoapi.flask_app] Exception on /jobs/b506cb48-9d8a-11ed-af5b-02420a000a06 [GET]
pygeoapi_pygeoapi.1.to5npjm7wwdq@server.dns    | Traceback (most recent call last):
pygeoapi_pygeoapi.1.to5npjm7wwdq@server.dns    |   File "/usr/lib/python3/dist-packages/flask/app.py", line 2070, in wsgi_app
pygeoapi_pygeoapi.1.to5npjm7wwdq@server.dns    |     response = self.full_dispatch_request()
pygeoapi_pygeoapi.1.to5npjm7wwdq@server.dns    |   File "/usr/lib/python3/dist-packages/flask/app.py", line 1515, in full_dispatch_request
pygeoapi_pygeoapi.1.to5npjm7wwdq@server.dns    |     rv = self.handle_user_exception(e)
pygeoapi_pygeoapi.1.to5npjm7wwdq@server.dns    |   File "/usr/lib/python3/dist-packages/flask/app.py", line 1513, in full_dispatch_request
pygeoapi_pygeoapi.1.to5npjm7wwdq@server.dns    |     rv = self.dispatch_request()
pygeoapi_pygeoapi.1.to5npjm7wwdq@server.dns    |   File "/usr/lib/python3/dist-packages/flask/app.py", line 1499, in dispatch_request
pygeoapi_pygeoapi.1.to5npjm7wwdq@server.dns    |     return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
pygeoapi_pygeoapi.1.to5npjm7wwdq@server.dns    |   File "/pygeoapi/pygeoapi/flask_app.py", line 361, in get_jobs
pygeoapi_pygeoapi.1.to5npjm7wwdq@server.dns    |     return get_response(api_.get_jobs(request, job_id))
pygeoapi_pygeoapi.1.to5npjm7wwdq@server.dns    |   File "/pygeoapi/pygeoapi/api.py", line 195, in inner
pygeoapi_pygeoapi.1.to5npjm7wwdq@server.dns    |     headers, status, content = func(*args, **kwargs)
pygeoapi_pygeoapi.1.to5npjm7wwdq@server.dns    |   File "/pygeoapi/pygeoapi/api.py", line 177, in inner
pygeoapi_pygeoapi.1.to5npjm7wwdq@server.dns    |     return func(cls, req_out, *args[2:])
pygeoapi_pygeoapi.1.to5npjm7wwdq@server.dns    |   File "/pygeoapi/pygeoapi/api.py", line 3244, in get_jobs
pygeoapi_pygeoapi.1.to5npjm7wwdq@server.dns    |     if JobStatus[job_['status']] in (
pygeoapi_pygeoapi.1.to5npjm7wwdq@server.dns    |   File "/usr/lib/python3.10/enum.py", line 440, in __getitem__
pygeoapi_pygeoapi.1.to5npjm7wwdq@server.dns    |     return cls._member_map_[name]
pygeoapi_pygeoapi.1.to5npjm7wwdq@server.dns    | KeyError: None
pygeoapi_pygeoapi.1.to5npjm7wwdq@server.dns    | KeyError: None

[TAlonglong]

Yeah, I set up retention time on redis results https://github.com/TAlonglong/satpy-pygeoapi-plugin/blob/main/satpy_pygeoapi_plugin/celery.py#L10

This should be able to config.

But also the manager need to handle this.

[epifanio]

Yes, we need the manager to return something that is not a 500 - will be nice to have a template page about excited jobs

[TAlonglong]

This is tricky as I get status pending from such a task. So I guess I ask the wrong way.

I need to dig in how.

[TAlonglong]

OK, I pushed a hack. Returning status dismissed for this give you valid response Is in main, so I guess you need to merge your modifications.