search

iotaledger / hive.go

A Go library containing data structures, various utils and abstractions which are used by both GoShimmer and Hornet.
Apache License 2.0
61 stars 24 forks source link

Data not persistent after restart in object storage #147

Closed lzpap closed 2 years ago

lzpap commented 4 years ago

Bug description

When shutting the the object storage (from goshimmer), something might not get properly persisted in the database. Upon restart, the node has a different number of messages in its storage then before restart.

Steps To reproduce the bug

  1. Run GoShimmer from feat/sync_revamp_inv branch. Possibly together with prometheus and grafana. Set metrics.local.db in the config file to true.
  2. Run the node for a couple of minutes. Observe tangle_solid_message_count exported metrics. (Or the printed values to console from here )
  3. Stop the node (gracefully)
  4. Start the node.
  5. tangle_solid_message_count exported metrics should have a different number than before the restart.

Expected behaviour

(Solid) message count in the object storage stays the same after restart.

Actual behaviour

(Solid) message count in the object storage randomly varies, depending on what machine you are running on. It might even be 0, or a greater value.

Note

According to @luca-moser , the root of the problem might be in the object storage, specifically, the batchwriter.

alexsporn commented 4 years ago

Verify you are calling Shutdown() on your ObjectStorage instances correctly when stopping the node

lzpap commented 4 years ago

@alexsporn Shutdown() is called, and interestingly the issue is more apparent on my local machine. It is harder to reproduce on my VPS. Maybe something related to how fast your disk is?

muXxer commented 2 years ago

@lzpap I have never seen this in hornet so far. Should we close that issue for now?