Atlas probe crawler crashed

[romain-fontugne]

Describe the bug The atlas probe crawler crashed today. The error seems to come from the code that convert the probe information dictionary into neo4j node attributes. This is the trace:

2023-11-08 08:46:47 MainProcess crawler crashed!!
Traceback (most recent call last):
  File "create_db.py", line 116, in <module>
    crawler.run()
  File "/home/romain/Projects/perso/internet-yellow-pages/iyp/crawlers/ripe/atlas_probes.py", line 124, in run
    probe_id = self.iyp.batch_create_nodes('AtlasProbe', 'id', flattened_probes)
  File "/home/romain/Projects/perso/internet-yellow-pages/iyp/__init__.py", line 319, in batch_create_nodes
    new_nodes = self.tx.run(create_query, batch=batch)
  File "/home/romain/Projects/perso/internet-yellow-pages/lib/python3.8/site-packages/neo4j/_sync/work/transaction
.py", line 156, in run
    result._tx_ready_run(query, parameters)
  File "/home/romain/Projects/perso/internet-yellow-pages/lib/python3.8/site-packages/neo4j/_sync/work/result.py",
 line 116, in _tx_ready_run
    self._run(query, parameters, None, None, None, None)
  File "/home/romain/Projects/perso/internet-yellow-pages/lib/python3.8/site-packages/neo4j/_sync/work/result.py",
 line 147, in _run
    self._connection.run(
  File "/home/romain/Projects/perso/internet-yellow-pages/lib/python3.8/site-packages/neo4j/_sync/io/_common.py", 
line 180, in inner
    func(*args, **kwargs)
  File "/home/romain/Projects/perso/internet-yellow-pages/lib/python3.8/site-packages/neo4j/_sync/io/_bolt5.py", l
ine 183, in run
    self._append(b"\x10", fields,
  File "/home/romain/Projects/perso/internet-yellow-pages/lib/python3.8/site-packages/neo4j/_sync/io/_bolt.py", li
ne 604, in _append
    self.outbox.append_message(signature, fields, dehydration_hooks)
  File "/home/romain/Projects/perso/internet-yellow-pages/lib/python3.8/site-packages/neo4j/_sync/io/_common.py", 
line 134, in append_message
    self._packer.pack_struct(tag, fields, dehydration_hooks)
  File "/home/romain/Projects/perso/internet-yellow-pages/lib/python3.8/site-packages/neo4j/_codec/packstream/v1/_
_init__.py", line 196, in pack_struct
    self.pack(field, dehydration_hooks=dehydration_hooks)
  File "/home/romain/Projects/perso/internet-yellow-pages/lib/python3.8/site-packages/neo4j/_codec/packstream/v1/_
_init__.py", line 111, in pack
    self.pack(item, dehydration_hooks=dehydration_hooks)
  File "/home/romain/Projects/perso/internet-yellow-pages/lib/python3.8/site-packages/neo4j/_codec/packstream/v1/_
_init__.py", line 100, in pack
    self.pack(item, dehydration_hooks=dehydration_hooks)
  File "/home/romain/Projects/perso/internet-yellow-pages/lib/python3.8/site-packages/neo4j/_codec/packstream/v1/_
_init__.py", line 111, in pack
    self.pack(item, dehydration_hooks=dehydration_hooks)
  File "/home/romain/Projects/perso/internet-yellow-pages/lib/python3.8/site-packages/neo4j/_codec/packstream/v1/_
_init__.py", line 121, in pack
    raise ValueError("Values of type %s are not supported" % type(value))
ValueError: Values of type <class 'flatdict.FlatterDict'> are not supported

[m-appel]

Thanks, I will check this out. It's a bit weird since the FlatterDict should be converted to a dict, but oh well.

[m-appel]

I can't seem to reproduce the issue and there should be no way for a FlatterDict to slip into the flattened_probes list, since they are converted to normal dicts here. Did this happen during our weekly dump?

[romain-fontugne]

yes, that happened today on the weekly run

[romain-fontugne]

is the probe data cached? Anyways when this run finishes I'll try to run it again on the same machine, let's see if the problem persists

[m-appel]

No, it is not cached since the query does not take that long. Okay, let's see what happens.

[romain-fontugne]

it worked! who knows what happened...

[m-appel]

Nice.