Open pcdavid opened 3 months ago
Locally, with 10 parallel threads I'm able to load a project with 1.2M elements (split across many resources) in about 2.5s, compared to about 15s when loading sequentially.
https://github.com/user-attachments/assets/814567a4-d2ef-4ba2-af27-ced65f968ecd
EditingContextPersistenceService.persist(IEditingContext)
could probably also be made to invoke .map(this.resourceToDocumentService::toDocument)
in parallel. Could this be as simple as calling .parallel()
on the stream of resources?threshold
(in number of documents) above which parallel loading/saving is enabled. The default could be -1
or 0
to disable the feature, and setting it to e.g. 5
would mean "only try to parallelize loading/saving of the project contains at least 5 resources/.documents"Executors.newFixedThreadPool(10)
).
Currently
org.eclipse.sirius.web.application.editingcontext.services.EditingContextSearchService.loadSemanticData(EditingContext, SemanticData)
loads all the Documents inside a project sequentially inside the target EMF ResourceSet.When a project's data is split across multiple documents, it is possible to load them in parallel, each in its own resource set, and only once they are all loaded to move them into the final, shared resource set.
Depending on how many documents there are and how many threads we assign to this, it can give a significant speedup (or not).