hashgraph / hedera-block-node

New Block Node services
Apache License 2.0
24 stars 3 forks source link

refactor: BlockAsDirReader and BlockAsDirWriter review and overhaul #319

Open ata-nas opened 1 month ago

ata-nas commented 1 month ago

Story

AS A BN Developer I WANT a review and overhaul if needed of the BlockAsDirReader and BlockAsDirWriter SO THAT concerns about current implementation are addressed and we are assured of their proper functioning.

Tech Notes

Based on this comment and the thread that proceeds it, there are a few concerns with the current implementation of the BlockAsDirReader and BlockAsDirWriter, mainly:

UPDATE:

AlfredoG87 commented 3 weeks ago

I don't think is worth it to continue to work on these implementations since we should remove them once the BlockAsFile Writer and Reader are completed.

ata-nas commented 2 weeks ago

I don't think is worth it to continue to work on these implementations since we should remove them once the BlockAsFile Writer and Reader are completed.

@AlfredoG87 during the implementation of #272 we reached the conclusion that we should revise these implementations here. It is true that if we shall delete them we should not work any longer over them, but my understanding is that the BlockAsDirReader and BlockAsDirWriter should NOT be deleted, @jsync-swirlds am I right about that?

jsync-swirlds commented 2 weeks ago

I don't think is worth it to continue to work on these implementations since we should remove them once the BlockAsFile Writer and Reader are completed.

There is no good reason to remove these implementations, as they're still useful for testing, debugging, and development. Certainly they are not production implementations, but at most I could see moving them to a testFixtures module (and I'm not certain we need to do that).

The persistence service should have multiple reader and writer implementations both to allow for flexibility in deployment and to ensure we continue to write the system with that required flexibility in mind. Example scenarios What if I want to deploy a block node that only stores the last few hours of blocks, without compression? What if I want a block node that stores the individual items on disk for another process to pick up?