Closed tangobravo62 closed 1 year ago
Cf. GitHub Issue https://github.com/dlr-eoc/prosEO/issues/162.
Alas improving/correcting the equals
and hashCode
methods did nothing to improve the performance substantially. In the end a performance improvement could be reached by not updating the product itself in the query, but just adding the product to the query's list of satisfying products (although this depends on the fact that the relationship from product to product query is not needed any more in the same transaction).
This was changed with commit b808580 and deployed with prosEO 0.9.1 in September 2022.
During the Sentinel-5P mission reprocessing, the releasing of job steps slowed down continually, and the memory requirements increased over time. At some point it was necessary to increase the brain RAM from 12 GiB to 24 GiB to ensure continuing production.
From the logs it was obvious that between 1 and 3 minutes per job step were spent in the ProductQueryService in the
executeQuery()
method. This happened during the selection of one of the auxiliary products, which are needed for every job step, and thus have a very large set of related ProductQueries. However, there are three such products for Sentinel-1B L1B processing, and it was always the first of them, which was referenced during the release process, but not always the same. The log file pattern looks like this:No matter in which order the three products were selected, it was always for the first one that the wait happened (and the memory usage of the Java VM increased), while for the other ones the wait was below 1 s.
The affected code in
ProductQueryService::executeQuery()
is the following:Although a full profiling has not yet taken place, the supposed culprit is the statement
product.getSatisfiedProductQueries().add(productQuery);
.