caracal-pipeline / crystalball

Distributed prediction of visibilities from a sky model
GNU General Public License v2.0
2 stars 5 forks source link

"ValueError: malformed node or string" when reading source list #42

Open o-smirnov opened 3 years ago

o-smirnov commented 3 years ago

Lucky me ended up with a non-parseable source list:

# running /home/oms/.venv/stimela/bin/crystalball --output-column DIR1_DATA --within dd.reg --num-sources 1000 --sky-model im4/im4-s08-sources.txt ../msdir/1608538564_sdp_l0-Jupiter-scan08.ms
# 2021-05-17 12:45:57 | INFO     | crystalball:_predict | Crystalball version 0.3.0
# 2021-05-17 12:45:57 | INFO     | region:load_regions | Read 1 inclusion region(s) from dd.reg
# Traceback (most recent call last):
#   File "/home/oms/.venv/stimela/bin/crystalball", line 33, in <module>
#     sys.exit(load_entry_point('crystalball', 'console_scripts', 'crystalball')())
#   File "/home/oms/projects/crystalball/crystalball/crystalball.py", line 155, in predict
#     return _predict(args)
#   File "/home/oms/projects/crystalball/crystalball/crystalball.py", line 172, in _predict
#     num=args.num_sources or None)
#   File "/home/oms/projects/crystalball/crystalball/wsclean.py", line 62, in import_from_wsclean
#     in load(wsclean_comp_list)}
#   File "/home/oms/.venv/stimela/lib/python3.6/site-packages/africanus/model/wsclean/file_model.py", line 208, in load
#     return _parse_lines(fh, line_nr, column_names, defaults, converters)
#   File "/home/oms/.venv/stimela/lib/python3.6/site-packages/africanus/model/wsclean/file_model.py", line 139, in _parse_lines
#     data_list.append(conv(value))
#   File "/usr/lib/python3.6/ast.py", line 85, in literal_eval
#     return _convert(node_or_string)
#   File "/usr/lib/python3.6/ast.py", line 61, in _convert
#     return list(map(_convert, node.elts))
#   File "/usr/lib/python3.6/ast.py", line 70, in _convert
#     operand = _convert(node.operand)
#   File "/usr/lib/python3.6/ast.py", line 84, in _convert
#     raise ValueError('malformed node or string: ' + repr(node))
# ValueError: malformed node or string: <_ast.Name object at 0x7fcd58f2e6a0>

@sjperkins you can get the source list at /net/muddy/home/oms/projects/Jove/selfcal2/im4/im4-s08-sources.txt (from any SARAO cluster node).

o-smirnov commented 3 years ago

Hmm actually, looks like it's because some of those sources were dumped as NANs:

s0c4162,POINT,-03:50:36.611,-20.37.04.578,-nan,[-nan,-nan,-nan],false,1337395507.8125,,,

Removing the nan-lines makes the list readable.

Maybe this is a wsclean bug (@francescaLoi and myself are currently discussing it with Andre), but it would be nice to add some robustness against it, particularly if Andre insists this sort of thing is legit...

sjperkins commented 3 years ago

Yep, this looks like it would need a fix in codex-africanus. I don't think I can transfer the issue over, so I'll just cut and paste your report over there.

sjperkins commented 3 years ago

While the change itself is not difficult, there are a number of steps that will need to take place.

codex-africanus has now moved forward to 3.7 as the minimum python version in https://github.com/ska-sa/codex-africanus/pull/248. Any fix for https://github.com/ska-sa/codex-africanus/issues/249 will require a codex release, as well as a Python 3.7 crystalball release.

None of these steps are difficult, but devs/users should be aware of the Python 3.7 upgrade requirement.