Unable to create tokenvm chains in a Docker setup

[Nuttymoon]

Describe the bug AvalancheGo fails to create tokenvm chains when running in Docker with a pebble error.

To Reproduce Follow the steps at ansible-avalanche-getting-started/docker

1. Build the ash/avalanche-node:1.10.12-tokenvm-0.0.15 Docker image
2. Start the local network with Docker compose
3. Create the Subnet using Ansible
4. Rename the dirs/files if the Subnet/chain IDs are different
5. Restart the local network
6. Check the main.log file of a node. You will see the following error at the creation of the chain:

   $ docker exec -it ash-avalanche-local-validator01 tail -f /var/log/avalanche/avalanchego/main.log
   ...
   [10-30|15:48:24.153] INFO chains/manager.go:330 creating chain {"subnetID": "29uVeLPJB1eQJkzRemU8g8wZDw5uJRqpab5U2mX9euieVwiEbL", "chainID": "2ArqB8j5FWQY9ZBtA3QFJgiH9EmXzbqGup5kuyPQZVZcL913Au", "vmID": "tHBYNu8ikqo4MWMHehC9iKB9mR5tB3DWzbkYmTfe9buWQ5GZ8"}
   [10-30|15:48:24.198] INFO chains/manager.go:1120 creating proposervm wrapper {"activationTime": "[12-05|05:00:00.000]", "minPChainHeight": 0, "minBlockDelay": "0s", "numHistoricalBlocks": 50000}
   [10-30|15:48:26.070] ERROR chains/manager.go:360 error creating chain {"subnetID": "29uVeLPJB1eQJkzRemU8g8wZDw5uJRqpab5U2mX9euieVwiEbL", "chainID": "2ArqB8j5FWQY9ZBtA3QFJgiH9EmXzbqGup5kuyPQZVZcL913Au", "chainAlias": "2ArqB8j5FWQY9ZBtA3QFJgiH9EmXzbqGup5kuyPQZVZcL913Au", "vmID": "tHBYNu8ikqo4MWMHehC9iKB9mR5tB3DWzbkYmTfe9buWQ5GZ8", "error": "error while creating new snowman vm rpc error: code = Unknown desc = Compact start  is not less than end "}
   [10-30|15:48:26.071] INFO health/worker.go:111 registered new check and initialized its state to failing {"namespace": "health", "name": "2ArqB8j5FWQY9ZBtA3QFJgiH9EmXzbqGup5kuyPQZVZcL913Au", "tags": ["29uVeLPJB1eQJkzRemU8g8wZDw5uJRqpab5U2mX9euieVwiEbL"]}
   ...

   The low level error is:

   error while creating new snowman vm rpc error: code = Unknown desc = Compact start  is not less than end 

   which seems to be a pebble error: https://github.com/cockroachdb/pebble/blob/ed45a776717595a33cce15c787e8583762c577a6/db.go#L1727

What is pretty surprising is that start and end are empty in the error message.

Expected behavior While creating a tokenvm chain fails in Docker, I am able to:

• Successfully create a tokenvm chain with the exact same versions and configuration in a virtualized environment (Ubuntu 22.04 on Multipass, AvalancheGo 1.10.12, tokenvm 0.0.15)
• Successfully create a subnet-evm chain in the same Docker setup (Ubuntu 22.04, AvalancheGo 1.10.12, subnet-evm 0.5.6)

Logs found in main.log for a tokenvm creation in the VM setup with similar versions and configuration:

$ multipass exec validator01 -- sudo cat /var/log/avalanche/avalanchego/main.log
[10-30|16:11:31.985] INFO chains/manager.go:330 creating chain {"subnetID": "29uVeLPJB1eQJkzRemU8g8wZDw5uJRqpab5U2mX9euieVwiEbL", "chainID": "2ArqB8j5FWQY9ZBtA3QFJgiH9EmXzbqGup5kuyPQZVZcL913Au", "vmID": "tHBYNu8ikqo4MWMHehC9iKB9mR5tB3DWzbkYmTfe9buWQ5GZ8"}
[10-30|16:11:32.013] INFO node/node.go:432 API server listening {"uri": "http://[::]:9650"}
[10-30|16:11:32.046] INFO chains/manager.go:1120 creating proposervm wrapper {"activationTime": "[12-05|05:00:00.000]", "minPChainHeight": 0, "minBlockDelay": "0s", "numHistoricalBlocks": 50000}
[10-30|16:11:32.369] INFO health/worker.go:111 registered new check and initialized its state to failing {"namespace": "health", "name": "2ArqB8j5FWQY9ZBtA3QFJgiH9EmXzbqGup5kuyPQZVZcL913Au", "tags": ["29uVeLPJB1eQJkzRemU8g8wZDw5uJRqpab5U2mX9euieVwiEbL"]}
[10-30|16:11:32.369] INFO server/server.go:229 adding route {"url": "/ext/bc/2ArqB8j5FWQY9ZBtA3QFJgiH9EmXzbqGup5kuyPQZVZcL913Au", "endpoint": "/coreapi"}
[10-30|16:11:32.369] INFO server/server.go:229 adding route {"url": "/ext/bc/2ArqB8j5FWQY9ZBtA3QFJgiH9EmXzbqGup5kuyPQZVZcL913Au", "endpoint": "/corews"}
[10-30|16:11:32.370] INFO server/server.go:229 adding route {"url": "/ext/bc/2ArqB8j5FWQY9ZBtA3QFJgiH9EmXzbqGup5kuyPQZVZcL913Au", "endpoint": "/tokenapi"}

Logs found in main.log for a subnet-evm chain creation in the same Docker setup:

$ docker exec -it ash-avalanche-local-validator01 tail -f /var/log/avalanche/avalanchego/main.log
[10-30|15:36:12.635] INFO chains/manager.go:330 creating chain {"subnetID": "29uVeLPJB1eQJkzRemU8g8wZDw5uJRqpab5U2mX9euieVwiEbL", "chainID": "RfX8YRUHePxFENBZFwaNt1tP6CR6RQ4hJJ5AqYv49LRoSKniK", "vmID": "srEXiWaHuhNyGwPUi444Tu47ZEDwxTWrbQiuD7FmgSAQ6X7Dy"}
[10-30|15:36:12.698] INFO chains/manager.go:1120 creating proposervm wrapper {"activationTime": "[12-05|05:00:00.000]", "minPChainHeight": 0, "minBlockDelay": "1s", "numHistoricalBlocks": 0}
[10-30|15:36:12.709] INFO health/worker.go:111 registered new check and initialized its state to failing {"namespace": "health", "name": "RfX8YRUHePxFENBZFwaNt1tP6CR6RQ4hJJ5AqYv49LRoSKniK", "tags": ["29uVeLPJB1eQJkzRemU8g8wZDw5uJRqpab5U2mX9euieVwiEbL"]}
[10-30|15:36:12.711] INFO server/server.go:229 adding route {"url": "/ext/bc/RfX8YRUHePxFENBZFwaNt1tP6CR6RQ4hJJ5AqYv49LRoSKniK", "endpoint": "/rpc"}
[10-30|15:36:12.711] INFO server/server.go:229 adding route {"url": "/ext/bc/RfX8YRUHePxFENBZFwaNt1tP6CR6RQ4hJJ5AqYv49LRoSKniK", "endpoint": "/ws"}

Logs

• main.log
• 2ArqB8j5FWQY9ZBtA3QFJgiH9EmXzbqGup5kuyPQZVZcL913Au.log

Operating System Ubuntu 22.04 (Docker) AvalancheGo 1.10.12 tokenvm 0.0.15

To best protect the Avalanche community security bugs should be reported in accordance to our Security Policy

[github-actions[bot]]

This issue has become stale because it has been open 60 days with no activity. Adding the lifecycle/frozen label will exempt this issue from future lifecycle events.