search

openmainframeproject / software-discovery-tool-deploy

View the live deployed version of SDT maintained by OMP
http://sdt.openmainframeproject.org/sdt/
Apache License 2.0
0 stars 7 forks source link

Failure to build cached_data.json on sdt.openmainframeproject.org #5

Closed pleia2 closed 2 years ago

pleia2 commented 2 years ago

A 500 error results due to the following error in the apache2 error.log:

[Mon Oct 25 15:14:35.783382 2021] [wsgi:error] [pid 10565:tid 139804448605952] [client 107.215.218.33:49960] Truncated or oversized response headers received from daemon process 'SDT': /opt/software-discovery-tool/src/SDT.wsgi, referer: http://sdt.openmainframeproject.org/

This occurs while attempting to create the initial cached_data.json file when the server reloads. As a temporary fix, I reduced the number of sources we're including in the tool (removed SUSE Package Hub and EoL Ubuntu releases). Ultimately, this will require some fine-tuning of Apache for better WSGI thread handling, a look at the code to optimize creation of the cache, or a larger VPS to work with.

rachejazz commented 2 years ago

It actually works now :') Did you do something?

rachejazz commented 2 years ago

https://github.com/openmainframeproject/software-discovery-tool/pull/72#issuecomment-951039098

Regarding this, yes! All json files are working now

pleia2 commented 2 years ago

Just reduced the number of sources! It seems to be too much load putting them all in the cached_data.json. Not sure yet what the failure point is, which we'll need to dig into.

pleia2 commented 2 years ago

Upgraded the server to the next tier:

Old: 1G memory, 1 shared CPU ($5/mo plan) New: 2G memory, 1 shared CPU ($10/mo plan)

The additional memory seems to have helped the file creation happen quickly enough so it doesn't fail. Also created the issue referenced above to tackle optimizing the creation of this file, since it's the only resource-intensive thing the tool does (day to day running of the tool is pretty low-resource).