[BUG] TypeError: required field "type_ignores" missing from Module

[BertrandBrelier]

Describe the bug Hello everybody, I would appreciate your help regarding the following issue:

I got the following error message when running more than one line of code in a jupyter cell on a remote Spark cluster:

An error was encountered: required field "type_ignores" missing from Module Traceback (most recent call last): File "/home/user/tmpdata/nm-local-dir/usercache/user/appcache/application_1644356194603_0003/container_1644356194603_0003_01_000001/tmp/3472037301374113689", line 223, in execute code = compile(mod, '', 'exec') TypeError: required field "type_ignores" missing from Module

To Reproduce run the following code in 1 jupyter cell:

print("5") print("6")

or

import numpy as np np.mean([1,2])

the code is running if each line is executed in a separate Jupyter Cell.

Expected behavior Expected to run the entire cell with multiple lines of code

Screenshots

Versions:

• SparkMagic: sparkmagic-0.19.1
• Livy (if you know it) : https://github.com/apache/incubator-livy
• Spark: spark-3.1.2
• Python : Python 3.9.7 (Anaconda3-2021.11-Linux-x86_64.sh)

Additional context Add any other context about the problem here.

Thank you for your help,

[BertrandBrelier]

This is actually a Livy issue impacted by the change of ast.Module requiring an additional argument in python 3.8: https://issues.apache.org/jira/browse/LIVY-795

Solution: update the fake_shell.py file from

for node in to_run_exec:
    mod = ast.Module([node])

To:

for node in to_run_exec:
    if sys.version_info >= (3,8):
          mod = ast.Module([node], type_ignores=[])
    else:
          mod = ast.Module([node])

Updating the fake_shell.py file and recompilling Livy fixed the issue.

Bertrand