Explorer does not have data of old heights

[noobvie]

Hello, I just built a new explorer https://scan.grin.money and the remaining issue is that the chart only show 3 recent days data and it only has the heights data last 3 days. I already enabled my node to archive mode. I wonder how can my node have data from genesis blocks like https://grinexplorer.net/? Is it possible to have your exported data in postgresql or your full blockchain directory data_chain?

Thank you.

[hendi]

Is it possible that you did the initial import ./manage.py import_from_tip before your archive node was fully synced, all the way back to genesis?

The import normally stops once it encounters a block it has already seen before. To avoid that and force it to go all the way back, use the --full-scan=True parameter: ./manage.py import_from_tip --full-scan=True

[noobvie]

Hi Hendi, Yes, I made sure that my node was fully synced and in running status (tried 2 times with pre-configured grin-server.toml with archive_mode = true) before I executed ./manage.py import_from_tip.

I've also tried to execute ./manage.py import_from_tip --full-scan=True and the last log like this:

Block 000650817176651b7f61b0612b0124734e862a13ac731b4a005f56357a2e7a8a already exists @ 1298884
Block 0000743d3be2cb3e8e3c463ab7be3ed6a5033b26bb5590b9af5f98ae68fd52d0 already exists @ 1298883
Block 0004ecb330d85202b41f39eb576307cb03aa6f9c03a4c6c4cffc5ab95f41375d already exists @ 1298882
Block 00040e0b86c207aebdf2bb24155241ca36f05b55bb4f9d3703169775d3f19805 already exists @ 1298881
Traceback (most recent call last):
  File "./grinexplorer/manage.py", line 15, in <module>
    execute_from_command_line(sys.argv)
  File "/home/grinviet/scan.grin.money/venv/lib/python3.8/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
    utility.execute()
  File "/home/grinviet/scan.grin.money/venv/lib/python3.8/site-packages/django/core/management/__init__.py", line 375, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/grinviet/scan.grin.money/venv/lib/python3.8/site-packages/django/core/management/base.py", line 323, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/home/grinviet/scan.grin.money/venv/lib/python3.8/site-packages/django/core/management/base.py", line 364, in execute
    output = self.handle(*args, **options)
  File "/home/grinviet/scan.grin.money/grinexplorer/blockchain/management/commands/import_from_tip.py", line 56, in handle
    (status, block_hash, prev_hash) = self.fetch_and_store_block(hash, parent)
  File "/home/grinviet/scan.grin.money/grinexplorer/blockchain/management/commands/import_from_tip.py", line 71, in fetch_and_store_block
    block_data = self.rpc("get_block", hash=hash, height=None, commit=None)
  File "/home/grinviet/scan.grin.money/grinexplorer/blockchain/management/commands/import_from_tip.py", line 28, in rpc
    return resp.json()["result"]["Ok"]
KeyError: 'Ok'

The line "Block 00040e0b86c207aebdf2bb24155241ca36f05b55bb4f9d3703169775d3f19805 already exists @ 1298881" is exactly the same my first synced height https://scan.grin.money/block/1298881 Thanks for your help.

[hendi]

Can you please edit the file blockchain/management/commands/import_from_tip.py, and add before line 28 (make sure to get the indentation right):

if "Ok" not in resp.json()["result"]:
    print(resp.json())
    exit()

return resp.json()["result"]["Ok"]  # <-- that was line 28 before

That should tell us what's wrong.

[noobvie]

I added the code:

Then run ./manage.py import_from_tip --full-scan=True again:

(venv) grinviet@scan:~/scan.grin.money$ python3 ./grinexplorer/manage.py import_from_tip --full-scan=True http://127.0.0.1:3413
Traceback (most recent call last):
  File "./grinexplorer/manage.py", line 15, in <module>
    execute_from_command_line(sys.argv)
  File "/home/grinviet/scan.grin.money/venv/lib/python3.8/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
    utility.execute()
  File "/home/grinviet/scan.grin.money/venv/lib/python3.8/site-packages/django/core/management/__init__.py", line 375, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/grinviet/scan.grin.money/venv/lib/python3.8/site-packages/django/core/management/__init__.py", line 224, in fetch_command
    klass = load_command_class(app_name, subcommand)
  File "/home/grinviet/scan.grin.money/venv/lib/python3.8/site-packages/django/core/management/__init__.py", line 36, in load_command_class
    module = import_module('%s.management.commands.%s' % (app_name, name))
  File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 844, in exec_module
  File "<frozen importlib._bootstrap_external>", line 981, in get_code
  File "<frozen importlib._bootstrap_external>", line 911, in source_to_code
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/grinviet/scan.grin.money/grinexplorer/blockchain/management/commands/import_from_tip.py", line 29
    exit()
    ^
IndentationError: unexpected indent

[hendi]

The indentation is wrong. The line with "print" must be indented exactly 4 spaces more than the previous line, and the line with "exit()" must have the same indentation as the print line. So based on your screenshot, the "print" line should have 4 spaces less, the "exit()" line 8 spaces less.

[noobvie]

It works again with different last log

Block 000639059caf6c5e1481cf5b2d3ac252f65dbfa05fe261e8faec07789b75f955 already exists @ 1298887
Block 00009c058cf7994ad3b8c5129923315202364a4ff9c312b3eae34d9dd7382afa already exists @ 1298886
Block 000630495aa5aa54962d1fb3070bda725872faff1734682a5d3dc64cdde6d4ea already exists @ 1298885
Block 000650817176651b7f61b0612b0124734e862a13ac731b4a005f56357a2e7a8a already exists @ 1298884
Block 0000743d3be2cb3e8e3c463ab7be3ed6a5033b26bb5590b9af5f98ae68fd52d0 already exists @ 1298883
Block 0004ecb330d85202b41f39eb576307cb03aa6f9c03a4c6c4cffc5ab95f41375d already exists @ 1298882
Block 00040e0b86c207aebdf2bb24155241ca36f05b55bb4f9d3703169775d3f19805 already exists @ 1298881
{'id': 'json', 'result': {'Err': 'NotFound'}}

[hendi]

This shows that the issue is your grin node, which doesn't have the block data for 1298881's parent (NotFound).

[noobvie]

Strange, could it be a 'bug' of grin node? I already set archive_mode = true and node is running pretty well.

[hendi]

IIRC correctly the "archive node" functionality is not present in 5.0.4 but only from 5.1.0 upwards. Right @antiochp ?

Can you try with a build from recent git master please? You might need to start from scratch with an empty chain_data.

[noobvie]

I built a new grin node version 5.2 alpha 1, completely new full synchronization is still in progress (over 12 hours passed). I will keep you updated on new test. Another thing, how big the current grin-explorer Postgresql database for storing data? I guess it's around 20GB? Maybe I will need another separately DB host in case.

[hendi]

Ok, sounds like it's working now, syncing a full archive nodes takes longer than a normal one.

$ du -sh /var/lib/postgresql/11/main/base/
14G     /var/lib/postgresql/11/main/base/

[noobvie]

Finally I could sync the old height to Postgresql. Thanks for your help. If you don't mind, I will enhance with more detailed information the file README.org later.