marg51 / foul

tracker based on elasticsearch
3 stars 0 forks source link

server: bulk operations should be smarter #2

Open marg51 opened 9 years ago

marg51 commented 9 years ago

draft

goal:

Improve performance and stability of /bulk endpoint, as well the architecture of the code. Readability is poor right now.

Stability:

Operations are currently not atomic and if one fails, it's reported as an error, as expected, but others will be successful. The integrity is compromised.

Each transaction should be saved in a log and kept if something went wrong

Performance:

We need to use the bulk endpoint from elasticsearch. It will be way faster than the current implementation. It's also easier to deal with errors. Since we generate IDs, everything can be flushed in the same transaction.

Architecture:

The code grew in size, let's put all managers into a new directory. server.coffee should have less logic in it and should only be a router.

marg51 commented 9 years ago

commits : 6b40502bb4f59c1ae49f95b56490cc0c6ae6a303 3b99697e85dadd74a09db055a0dd37a94733e999