Performance issues - Githubissues

dzierzy commented 7 years ago

It's needed to optimize notice processing in order to speed up operations.

[x] RESTfull services - single RestTemplate object is enough to process all calls
[x] JSON parsing - optimize amount of JSONObject instances
[x] multithreading - engine should be able to process notices in more than single thread. When we enabled 2 threads in a configuration, after processing around 50 000 notices there was constraint violation error reported when running helper db script as some identifiers where repeated. Looks like somewhere there are not synchronized resources. @juzraai did you try to process big amount of notices paralelly?

juzraai commented 7 years ago

did you try to process big amount of notices paralelly?

Of course I did. And I always use multithreading when I'm not testing. For example, today I parsed 360K notices using 4 threads without any error.

dzierzy commented 7 years ago

Seems like I had some remains in db so id conflict occured. Successfully process 100k items using 8 threads

petabyte-research / redflags