Import of problem zip slow with many submission files

[eldering]

Importing example_problems/hello as a zip takes about 10 seconds, and almost all of that time seems spent importing the submissions.

I think the problem is the double for loop over the files in the zip at https://github.com/DOMjudge/domjudge/blob/01b754ab46bd7195cc412313f559ccb7ed03f86c/webapp/src/Service/ImportProblemService.php#L635-L659, which may become slow when there's in the case of hello 223 files/directories in the zip file.

[meisterT]

I tried to investigate a bit, the double loop doesn't seem to be an issue. Also we only entering the inner loop for multi-file submissions - for the hello problem we only have a few of these. Commenting that code made it barely faster.

[meisterT]

Elaborating here a little bit more: the slow part is that we create all judging runs and judge tasks (one per test case and judging).

We discussed in Luxor that it we could have a special queuetask to do that instead per judging and let the judgedaemon create the actual judging_run and judge tasks for this submission.

If we do so, also rejudgings should be a lot faster.

[meisterT]

I looked into this again, and it gets a lot faster (before 21s, after 4s) if I disable the event log here: https://github.com/DOMjudge/domjudge/blob/main/webapp/src/Service/SubmissionService.php#L637-L641

Slow part in there is the internal API request (goes up from 4s to 5s if I do the rest but not the internal API request): https://github.com/DOMjudge/domjudge/blob/main/webapp/src/Service/EventLogService.php#L319

The interesting part seems to be that each of these requests seems to take longer, they take around 5-20ms for the first few requests and can go up to 200ms and more per request:

Now I first thought that this increases with number of submissions in the database, but that doesn't seem to be true. Perhaps there is some other inefficiency in the stacked subrequests that grows with the number of subrequests within one parent request?

[nickygerritsen]

Did you run this in prod mode? In dev mode queries are logged and the more queries you do the slower this gets.

[meisterT]

No, I have used dev mode and also enabled the profiler, otherwise I would be blind. But that's a sensible explanation, I'll try later with prod mode and see how overall timings change.

[nickygerritsen]

You can try $em->getConnection()->getConfiguration()->setSQLLogger(null); to disable query logging at the beginning of the submission service.

[meisterT]

$this->em->getConnection()->getConfiguration()->setSQLLogger(null); did not change anything.

In prod mode the problem takes with N submissions:

• N=90: ~4s (⌀ 44ms)
• N=190: ~13.5s (⌀ 71ms)
• N=290: ~29.5s (⌀ 101ms)
• N=390: ~56.5s (⌀ 145ms)

There is clearly a component in there that grows the work we have to do per submission with the number of submissions.

Without doing any of the event logging (and thus internal API) the timings look a little bit better, but there is still the same overall behavior:

• N=90: ~3.3s (⌀ 37ms)
• N=190: ~11s (⌀ 58ms)
• N=290: ~24s (⌀ 83ms)
• N=390: ~42s (⌀ 108ms)

[nickygerritsen]

We should maybe check this with Blackfire to see where the time goes.