No report exists for key...

[dale-wahl]

I am currently moving a Flask app to Gunicorn and am having an issue where I cannot ever retrieve a report even though it appears that the command will eventually be successfully run. I will send a post to an API endpoint and will receive a successful response that includes the key and result_url, but when I make a get request to that url, I only receive the "No report exists for key..." response. At first I thought it might be a delay, but even with repeated get requests I am not able to receive reports.

I have been looking through your documentation to get a better idea of how to access reports and understand how long before they expire/disappear. I had noticed previously that if I keep making a request for a specific key, it will eventually return No report exists for key even if one had previously existed. I am wondering if you have run into similar issues and if you could point me in the right direction to try to track down these reports or log them somewhere more permanent. I also tried playing with the Executor a bit, but am not entirely sure where to start with that.

Thank you for the help.

[eshaan7]

Hi, @dale-wahl. Could you tell the python, flask and flask-shell2http versions your app is using ? Thanks.

[eshaan7]

The flask-shell2http logs at various requests/operations which could help you track this further. You can setup logging similar to how it's done in the examples.

[eshaan7]

Inactive.

[dale-wahl]

Hey eshaan7, sorry for the super long delay. I was quite distracted by another project and unable to come back to this until now.

I wanted to thank you for the logging info; it was quite easy to set up. I had much more trouble getting the logging for Gunicorn up and running (which was effecting your logging). Once I did, I figured out that the issue has to do with having multiple Gunicorn workers handling requests. The report only exists for the same worker that handled the original request. I am not sure how to solve it yet, but there you have it. Just did not want to leave you hanging.

Version information you requested: Python 3.7.11

Flask==2.0.2
Flask-Executor==0.9.4
Flask-Shell2HTTP==1.8.0  

[eshaan7]

Thank you for getting back to me. That's a good find. In my case, I never tried using flask-shell2http with multiple gunicorn workers but I can easily see why it messes things up - flask_shell2http and flask_executor both rely on global variables for state management which are of course not shared between different workers.

As of right now, I don't know if there is a nice way to share state between multiple gunicorn workers so I can only suggest 2 workarounds:

• Use a single gunicorn worker
• Gunicorn can work with gevent to create a server that can support multiple clients within a single worker process using coroutines.

[dale-wahl]

Thanks! I'll check out gevent and see if that will work for my case.

Since the process indeed runs, I may be able to infer or check if completes in some other way. Or, you know, bombard myself with requests until I get the correct worker haha.

[eshaan7]

I may be able to infer or check if completes in some other way

One solution I can recommend regarding this is:

• Use a callback function to store the result in a shared* database/cache that is accessible to all gunicorn workers. Best practice would be to use something robust like Redis for this but if you don't want that, you can also use something like python's inbuilt shelve module to persist key-value (dict type) objects to the file system.
• Create another route to query this central storage.

* Flask-shell2http provides attaching callback functions that get executed after the future completes. See this example.

EDIT:

multiprocessing.shared_memory might work too for central sharable storage.

[lingfish]

• Use a callback function* to store the result in a shared database/cache that is accessible to all gunicorn workers

This is nice, but is obviously only called after a (long running) future completes; what about when trying to poll for a key/job whilst it's still running? That too would need to go into Redis, but I can't figure a clean way to do this, when using another route to check Redis/the job.

multiprocessing.shared_memory might work too for central sharable storage.

This got me excited, and then I noticed it's >= Python 3.8, and the server I have to use isn't :(