initial application is done. can now insert events from xz compressed tarballs to timescaledb. insert rate is approx 8500msgs/sec -> ~6-8 times faster than real world scenario. but this is still very slow if we want to import years of data (will take months to insert)
TODO:
change the target to be pulsar instead of timescaledb
some minor error handling tweaks
Large-enough storage needed (Azure?)
TODO design:
is fast enough?
can we insert in parallel? is there data consistency issues?
how to do the "hotswap" part? First import all history data to some topic and then swap pulsar to insert fresh data into the same topic?
running the data into timescaleDB from Pulsar will take roughly the same amount of time than inserting it into pulsar in the first place but gladly those can be run almost in parallel (consumer to sync almost immediately)
initial application is done. can now insert events from xz compressed tarballs to timescaledb. insert rate is approx 8500msgs/sec -> ~6-8 times faster than real world scenario. but this is still very slow if we want to import years of data (will take months to insert)
TODO:
TODO design: