Do not block serving requests when running the JAR file

[kvbik]

The problem is, that in the web processing time they are opening a subprocess. You have quite a limited number of python server processes (some python wsgi server - gunicorn in our production case), let's say 10 per instance. In case of high web server load (eg.: announcing the conference), the site will be serving a lot of visitors. If at the same time people are also booking the tickets, they can block all of the web server processes! The java subprocess can take some time (microseconds if not seconds) and the web server cannot serve anything else when waiting for the java code to finish.

We need to either make it asynchronous or move it to some place where it does not block the server time that instant. Creating an order can happen way too often and could block the server.

Few ideas:

1. do the expensive computation (subprocess calling *.jar exec) in the email template
2. create a background task from it (using celery - django background process - probably used inside pretix, too)
3. ???

There are of course implications from any of the possible solutions.

[a5net]

After some research, it also felt like going with celery is the right option. The solution I was considering was running celery with putting a limit on simultaneous tasks (so we won't create too many processes). However, I am not experienced with celery, so I am not sure what kind of problems this solution might have.

[kvbik]

I have a lot of experience with celery. The problem here is the asynchronicity - you'd have to have some placeholders before it is filled with the right content. And the email could be sent before we have the proper information...

[ligi]

@ayanginet did you start working on this? We can actually control when the email goes out if this is a problem by using the placeholder in "send out mails"

[a5net]

Looks like to me that the case when Attestation Link is not present is already handled here: https://github.com/efdevcon/pretix-attestation-placeholder-plugin/blob/38c6918b5e61770db19b58b516661046c134f7cd/pretix_attestation_plugin/email.py#L46-L47

And the object is not created unless we have the link: https://github.com/efdevcon/pretix-attestation-placeholder-plugin/blob/38c6918b5e61770db19b58b516661046c134f7cd/pretix_attestation_plugin/signals.py#L56-L64 So I think it's enough just to convert the current link generation process into a celery job. What do you think @ligi @kvbik ?

[ligi]

Yea - it's handled in the sense there is no crash - but users will not get the attestation link and we will have a lot of support cases ;-) We discussed in a call yesterday and think the best solution is to move the generation of the link from the time of the order to the sending of the email - what do you think?

[a5net]

Yea - it's handled in the sense there is no crash - but users will not get the attestation link and we will have a lot of support cases ;-) We discussed in a call yesterday and think the best solution is to move the generation of the link from the time of the order to the sending of the email - what do you think?

Just to make sure that I understood correctly, the solution you proposing is to move the link generation part to email rendering, without any background celery jobs? If yes, then doesn't it mean the person who is sending the email should wait until all the links are generated?

[ligi]

yes you understood correctly - but if we move the link generation to the email rendering there should be no waiting of humans for links as far as I see

[ligi]

closed by #39