search

AustralianSynchrotron / lightflow

A lightweight, distributed workflow system
https://australiansynchrotron.github.io/lightflow/
BSD 3-Clause "New" or "Revised" License
100 stars 18 forks source link

Strange AttributeError #15

Open jrmlhermitte opened 6 years ago

jrmlhermitte commented 6 years ago

I'm running a fairly complicated pipeline but it works surprisingly well. ( this dag calls this dag which calls this dag )

However, sometimes, I encounter this error below on the DAG. All the tasks, however, complete just fine.

Any idea what this may come from? Have you seen this before? Any tips on debugging? I will also keep searching myself but I thought I'd flag this.

Traceback (most recent call last):
  File "/home/xf11bm/miniconda3/envs/lightflow-pipeline/lib/python3.6/site-packages/kombu/serialization.py", line 50, in _reraise_errors
    yield
  File "/home/xf11bm/miniconda3/envs/lightflow-pipeline/lib/python3.6/site-packages/kombu/serialization.py", line 263, in loads
    return decode(data)
  File "/home/xf11bm/projects/lightflow/lightflow/queue/pickle.py", line 15, in cloudpickle_loads
    return load(BytesIO(s))
AttributeError: type object 'int64' has no attribute '__index__'

dagerrorissue

jrmlhermitte commented 6 years ago

Just to add, when I re-run this exact pipeline, it seems to be the same dataset. I will try to use the remote debugger a little bit when i have time. Will have to binary search...