search

TrueBlocks / trueblocks-core

The main repository for the TrueBlocks system
https://trueblocks.io
GNU General Public License v3.0
1.05k stars 198 forks source link

Feature/handle sigterm #3730

Closed dszlachta closed 3 months ago

dszlachta commented 4 months ago

This changes the scraper code, so that when user presses Ctrl-C, we print a message, finish the current job and exit. If the code is in a loop, it will break it.

The idea is to use the cleanup mechanism we already have.

dszlachta commented 3 months ago

@tjayrush I fixed the code accordingly to your feedback. There's now a new class of error, criticalError and it's returned from Consolidate if chunk write fails. When we get it, we remove both index and bloom files and then do the same thing like when a user hits Ctrl-C: break from the loop, cleanup and exit.

tjayrush commented 3 months ago

I'll review it and merge it tomorrow morning.

In the meantime, this is the mechanism I was talking about where a backup file is created and replaces the failed write: https://github.com/TrueBlocks/trueblocks-core/blob/master/src/apps/chifra/pkg/index/chunk_write.go#L83.

Also, I was worried that we were only returning criticalError when writing the index data, but this line looks like it would also fail when writing the Bloom filter: https://github.com/TrueBlocks/trueblocks-core/blob/master/src/apps/chifra/pkg/index/chunk_write.go#L122

Any error from chunk.Write will cause a criticalError, I think. Is that right?

dszlachta commented 3 months ago

Yes, that's correct, any error from chunk.Write will cause a criticalError.

I've seen the backup mechanism you mentioned. Let's keep in mind that it's there.