There are many issues in the production environment.

[sanjinhub]

If the data reaches one hundred million, many problems arise:

1. ConstructDatabase takes 1-2 hours to generate the data.
2. ConstructDatabase requires the input parameter block.binpb; even without block data, it still needs to be input.
3. PIRProcessDatabase takes 1-2 hours to generate.
4. PIRService requires loading the data into memory at startup, which takes more than 1 hour.
5. Since PIRService loads the data into memory at startup, it requires high-memory servers, which increases the cost.
6. Each update takes at least 5 hours, making the maintenance cost very high, and the updates cumbersome.
7. Although block data is written, it doesn't actually intercept, and this feature is ineffective.

Suggestions:

1. Please consider the design from the perspective of actual users.
2. The functionality is actually very simple, so please avoid making it too complex. Using MySQL/SQLite as the database would make updates and maintenance easier.

[fboemer]

Thanks for the feedback.

In case you haven't seen it, the Encoding Pipeline documentation has some tips on how to scale the pre-processing.

See also the dataset processing instructions, which give some suggested sharding parameters. For larger datasets, you may wish to load only a subset of the shards into a single server.