Could not compile on 16.04.5 LTS Xenial Xerus (Was: How to transcompile python code that have logging or psutil libraries imported.)

nutant-h commented 6 years ago

I am trying to transcompile a Python file into Go.

Following the README file I installed grumpy, then tested it; #echo 'import sys; print sys.version' | grumprun 2.7.13 works fine.

My python code needs logging and psutil libraries imported; but executing echo 'import logging, psutil, sys; print sys.version' | grumprun throws the following error.

# echo 'import logging, psutil, sys; print sys.version' | grumprun
Traceback (most recent call last):
  File "/root/grumpy/grumpy-runtime-src/build/bin/grumprun", line 8, in <module>
    sys.exit(cli.main())
  File "/usr/local/lib/python2.7/dist-packages/Click-7.0-py2.7.egg/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/Click-7.0-py2.7.egg/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python2.7/dist-packages/Click-7.0-py2.7.egg/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python2.7/dist-packages/Click-7.0-py2.7.egg/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python2.7/dist-packages/Click-7.0-py2.7.egg/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/root/grumpy/grumpy-tools-src/grumpy_tools/cli.py", line 84, in run
    result = grumprun.main(stream=stream, modname=modname, pep3147=pep3147, clean_tempfolder=(not keep_main))
  File "/root/grumpy/grumpy-tools-src/grumpy_tools/grumprun.py", line 114, in main
    transpiled = grumpc.main(stream, modname=modname, pep3147=True, recursive=True)
  File "/root/grumpy/grumpy-tools-src/grumpy_tools/grumpc.py", line 166, in main
    _recursively_transpile(import_objects)
  File "/root/grumpy/grumpy-tools-src/grumpy_tools/grumpc.py", line 110, in _recursively_transpile
    main(stream=open(imp_obj.script), modname=name, pep3147=True, recursive=True, return_result=False)
  File "/root/grumpy/grumpy-tools-src/grumpy_tools/grumpc.py", line 166, in main
    _recursively_transpile(import_objects)
  File "/root/grumpy/grumpy-tools-src/grumpy_tools/grumpc.py", line 110, in _recursively_transpile
    main(stream=open(imp_obj.script), modname=name, pep3147=True, recursive=True, return_result=False)
  File "/root/grumpy/grumpy-tools-src/grumpy_tools/grumpc.py", line 156, in main
    deps, import_objects = _collect_deps(script, modname, pep3147_folders, from_cache=(not will_refresh))
  File "/root/grumpy/grumpy-tools-src/grumpy_tools/grumpc.py", line 77, in _collect_deps
    deps, import_objects = pydeps.main(script, modname, with_imports=True) #, script, gopath)
  File "/root/grumpy/grumpy-tools-src/grumpy_tools/pydeps.py", line 35, in main
    imports = imputil.collect_imports(modname, script, gopath, package_dir=package_dir)
  File "/root/grumpy/grumpy-tools-src/grumpy_tools/compiler/imputil.py", line 216, in collect_imports
    collector.visit(mod)
  File "/root/grumpy/grumpy-tools-src/grumpy_tools/vendor/pythonparser/algorithm.py", line 41, in visit
    return self._visit_one(obj)
  File "/root/grumpy/grumpy-tools-src/grumpy_tools/vendor/pythonparser/algorithm.py", line 34, in _visit_one
    return self.generic_visit(node)
  File "/root/grumpy/grumpy-tools-src/grumpy_tools/vendor/pythonparser/algorithm.py", line 27, in generic_visit
    self.visit(getattr(node, field_name))
  File "/root/grumpy/grumpy-tools-src/grumpy_tools/vendor/pythonparser/algorithm.py", line 39, in visit
    return [self.visit(elt) for elt in obj]
  File "/root/grumpy/grumpy-tools-src/grumpy_tools/vendor/pythonparser/algorithm.py", line 41, in visit
    return self._visit_one(obj)
  File "/root/grumpy/grumpy-tools-src/grumpy_tools/vendor/pythonparser/algorithm.py", line 34, in _visit_one
    return self.generic_visit(node)
  File "/root/grumpy/grumpy-tools-src/grumpy_tools/vendor/pythonparser/algorithm.py", line 27, in generic_visit
    self.visit(getattr(node, field_name))
  File "/root/grumpy/grumpy-tools-src/grumpy_tools/vendor/pythonparser/algorithm.py", line 39, in visit
    return [self.visit(elt) for elt in obj]
  File "/root/grumpy/grumpy-tools-src/grumpy_tools/vendor/pythonparser/algorithm.py", line 41, in visit
    return self._visit_one(obj)
  File "/root/grumpy/grumpy-tools-src/grumpy_tools/vendor/pythonparser/algorithm.py", line 32, in _visit_one
    return getattr(self, visit_attr)(node)
  File "/root/grumpy/grumpy-tools-src/grumpy_tools/compiler/imputil.py", line 205, in visit_ImportFrom
    self.imports.extend(self.importer.visit(node))
  File "/root/grumpy/grumpy-tools-src/grumpy_tools/vendor/pythonparser/algorithm.py", line 41, in visit
    return self._visit_one(obj)
  File "/root/grumpy/grumpy-tools-src/grumpy_tools/vendor/pythonparser/algorithm.py", line 32, in _visit_one
    return getattr(self, visit_attr)(node)
  File "/root/grumpy/grumpy-tools-src/grumpy_tools/compiler/imputil.py", line 119, in visit_ImportFrom
    imp = self._resolve_import(node, node.module)
  File "/root/grumpy/grumpy-tools-src/grumpy_tools/compiler/imputil.py", line 170, in _resolve_import
    raise util.ImportError(node, 'no such module: {} (script: {})'.format(modname, self.script))
grumpy_tools.compiler.util.ImportError: line 15: no such module: _codecs (script: /usr/lib/python2.7/codecs.py)

I am sure there must be a way to use other libraries. How can I use logging library? I am just not able to find the documentations, need help.

Thank you.

nutant-h commented 6 years ago

Found out this https://github.com/google/grumpy/wiki/Standard-libraries:-guidelines-for-integration

nutant-h commented 6 years ago

Followed the above mentioned wiki. though getting the same error.

root@dev-vm-charlie:~/grumpy/grumpy-runtime-src# make
find: ‘compiler’: No such file or directory
find: ‘compiler’: No such file or directory
find: ‘compiler’: No such file or directory
make: *** No rule to make target 'build/pkg/linux_amd64/__python__/codecs.a', needed by 'build/pkg/linux_amd64/__python__/logging.a'.  Stop.
root@dev-vm-charlie:~/grumpy/grumpy-runtime-src# echo 'import sys; print sys.version' | grumprun
2.7.13
root@dev-vm-charlie:~/grumpy/grumpy-runtime-src# echo 'import logging, sys; print sys.version' | grumprun
Traceback (most recent call last):
  File "/root/grumpy/grumpy-runtime-src/build/bin/grumprun", line 8, in <module>
    sys.exit(cli.main())
  File "/usr/local/lib/python2.7/dist-packages/Click-7.0-py2.7.egg/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/Click-7.0-py2.7.egg/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python2.7/dist-packages/Click-7.0-py2.7.egg/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python2.7/dist-packages/Click-7.0-py2.7.egg/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python2.7/dist-packages/Click-7.0-py2.7.egg/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/root/grumpy/grumpy-tools-src/grumpy_tools/cli.py", line 84, in run
    result = grumprun.main(stream=stream, modname=modname, pep3147=pep3147, clean_tempfolder=(not keep_main))
  File "/root/grumpy/grumpy-tools-src/grumpy_tools/grumprun.py", line 114, in main
    transpiled = grumpc.main(stream, modname=modname, pep3147=True, recursive=True)
  File "/root/grumpy/grumpy-tools-src/grumpy_tools/grumpc.py", line 166, in main
    _recursively_transpile(import_objects)
  File "/root/grumpy/grumpy-tools-src/grumpy_tools/grumpc.py", line 110, in _recursively_transpile
    main(stream=open(imp_obj.script), modname=name, pep3147=True, recursive=True, return_result=False)
  File "/root/grumpy/grumpy-tools-src/grumpy_tools/grumpc.py", line 166, in main
    _recursively_transpile(import_objects)
  File "/root/grumpy/grumpy-tools-src/grumpy_tools/grumpc.py", line 110, in _recursively_transpile
    main(stream=open(imp_obj.script), modname=name, pep3147=True, recursive=True, return_result=False)
  File "/root/grumpy/grumpy-tools-src/grumpy_tools/grumpc.py", line 156, in main
    deps, import_objects = _collect_deps(script, modname, pep3147_folders, from_cache=(not will_refresh))
  File "/root/grumpy/grumpy-tools-src/grumpy_tools/grumpc.py", line 77, in _collect_deps
    deps, import_objects = pydeps.main(script, modname, with_imports=True) #, script, gopath)
  File "/root/grumpy/grumpy-tools-src/grumpy_tools/pydeps.py", line 35, in main
    imports = imputil.collect_imports(modname, script, gopath, package_dir=package_dir)
  File "/root/grumpy/grumpy-tools-src/grumpy_tools/compiler/imputil.py", line 216, in collect_imports
    collector.visit(mod)
  File "/root/grumpy/grumpy-tools-src/grumpy_tools/vendor/pythonparser/algorithm.py", line 41, in visit
    return self._visit_one(obj)
  File "/root/grumpy/grumpy-tools-src/grumpy_tools/vendor/pythonparser/algorithm.py", line 34, in _visit_one
    return self.generic_visit(node)
  File "/root/grumpy/grumpy-tools-src/grumpy_tools/vendor/pythonparser/algorithm.py", line 27, in generic_visit
    self.visit(getattr(node, field_name))
  File "/root/grumpy/grumpy-tools-src/grumpy_tools/vendor/pythonparser/algorithm.py", line 39, in visit
    return [self.visit(elt) for elt in obj]
  File "/root/grumpy/grumpy-tools-src/grumpy_tools/vendor/pythonparser/algorithm.py", line 41, in visit
    return self._visit_one(obj)
  File "/root/grumpy/grumpy-tools-src/grumpy_tools/vendor/pythonparser/algorithm.py", line 34, in _visit_one
    return self.generic_visit(node)
  File "/root/grumpy/grumpy-tools-src/grumpy_tools/vendor/pythonparser/algorithm.py", line 27, in generic_visit
    self.visit(getattr(node, field_name))
  File "/root/grumpy/grumpy-tools-src/grumpy_tools/vendor/pythonparser/algorithm.py", line 39, in visit
    return [self.visit(elt) for elt in obj]
  File "/root/grumpy/grumpy-tools-src/grumpy_tools/vendor/pythonparser/algorithm.py", line 41, in visit
    return self._visit_one(obj)
  File "/root/grumpy/grumpy-tools-src/grumpy_tools/vendor/pythonparser/algorithm.py", line 32, in _visit_one
    return getattr(self, visit_attr)(node)
  File "/root/grumpy/grumpy-tools-src/grumpy_tools/compiler/imputil.py", line 205, in visit_ImportFrom
    self.imports.extend(self.importer.visit(node))
  File "/root/grumpy/grumpy-tools-src/grumpy_tools/vendor/pythonparser/algorithm.py", line 41, in visit
    return self._visit_one(obj)
  File "/root/grumpy/grumpy-tools-src/grumpy_tools/vendor/pythonparser/algorithm.py", line 32, in _visit_one
    return getattr(self, visit_attr)(node)
  File "/root/grumpy/grumpy-tools-src/grumpy_tools/compiler/imputil.py", line 119, in visit_ImportFrom
    imp = self._resolve_import(node, node.module)
  File "/root/grumpy/grumpy-tools-src/grumpy_tools/compiler/imputil.py", line 170, in _resolve_import
    raise util.ImportError(node, 'no such module: {} (script: {})'.format(modname, self.script))
grumpy_tools.compiler.util.ImportError: line 15: no such module: _codecs (script: /usr/lib/python2.7/codecs.py)

alanjds commented 6 years ago

Hi, @nutant-h

The README is outdated. I should update it. Sorry for that.

Please follow the following (copied from https://github.com/grumpyhome/grumpy/pull/79#issue-216417478):

Install Python 2.7
Install Go >= 1.10

Install Grumpy packages for development:


cd grumpy-tools-src/
python setup.py develop

cd grumpy-runtime-src/ python setup.py develop

- Run your tests via the `grumpy` command line tool:
```bash
$ grumpy -c 'import sys; print sys.version'
2.7.13

This should be working.

alanjds commented 6 years ago

I created an issue to track the update of README on #113. Thanks for reporting.

Please feed back if my instructions had worked or not. Are the instructions clear enough?

nutant-h commented 6 years ago

Hi @alanjds Followed your instructions;

I got error while doing

cd grumpy-runtime-src/
python setup.py develop

This is the error;

root@dev-vm-charlie:~/grumpy/grumpy-runtime-src# python setup.py develop
/usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'long_description_content_type'
  warnings.warn(msg)
/usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'project_urls'
  warnings.warn(msg)
/usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'python_requires'
  warnings.warn(msg)

Installed /root/grumpy/grumpy-runtime-src/.eggs/setuptools-40.4.3-py2.7.egg
Traceback (most recent call last):
  File "setup.py", line 174, in <module>
    setup(**GRUMPY_RUNTIME_OPTIONS)
  File "/usr/lib/python2.7/distutils/core.py", line 111, in setup
    _setup_distribution = dist = klass(attrs)
  File "/usr/lib/python2.7/dist-packages/setuptools/dist.py", line 272, in __init__
    _Distribution.__init__(self,attrs)
  File "/usr/lib/python2.7/distutils/dist.py", line 287, in __init__
    self.finalize_options()
  File "/usr/lib/python2.7/dist-packages/setuptools/dist.py", line 327, in finalize_options
    ep.load()(self, ep.name, value)
  File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 2229, in load
    return self.resolve()
  File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 2239, in resolve
    raise ImportError(str(exc))
ImportError: 'module' object has no attribute 'check_specifier'

alanjds commented 6 years ago

This is strange, as we have automated tests for Linux. A couple of questions:

What version/distribution of Linux you are using?
What version of Python (python --version)? Installed via apt/yum?
The error persists if trying in a virtualenv?

I am trying to reproduce here, but it works for me

nutant-h commented 6 years ago

Hi

What version/distribution of Linux you are using?

root@dev-vm-charlie:~# cat /etc/os-release
NAME="Ubuntu"
VERSION="16.04.5 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.5 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial

What version of Python (python --version)? Installed via apt/yum?

root@dev-vm-charlie:~# python --version
Python 2.7.12

The error persists if trying in a virtualenv? Have not tried that but how would it make any difference?

nutant-h commented 6 years ago

I tried in virtualenv and it worked till the point where I did

cd grumpy-runtime-src/
python setup.py develop

and at the end got this;

    File 'build/pkg/linux_amd64/__python__/logging.a' does not exist.
       File 'build/src/__python__/logging/module.go' does not exist.
      Must remake target 'build/src/__python__/logging/module.go'.
      Successfully remade target file 'build/src/__python__/logging/module.go'.
       File 'build/pkg/linux_amd64/__python__/codecs.a' does not exist.
      Must remake target 'build/pkg/linux_amd64/__python__/codecs.a'.
make: *** No rule to make target 'build/pkg/linux_amd64/__python__/codecs.a', needed by 'build/pkg/linux_amd64/__python__/logging.a'.  Stop.
Traceback (most recent call last):
  File "setup.py", line 174, in <module>
    setup(**GRUMPY_RUNTIME_OPTIONS)
  File "/root/grumpy2/local/lib/python2.7/site-packages/setuptools/__init__.py", line 140, in setup
    return distutils.core.setup(**attrs)
  File "/usr/lib/python2.7/distutils/core.py", line 151, in setup
    dist.run_commands()
  File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/root/grumpy2/local/lib/python2.7/site-packages/setuptools/command/develop.py", line 38, in run
    self.install_for_development()
  File "/root/grumpy2/local/lib/python2.7/site-packages/setuptools/command/develop.py", line 138, in install_for_development
    self.run_command('build_ext')
  File "/usr/lib/python2.7/distutils/cmd.py", line 326, in run_command
    self.distribution.run_command(command)
  File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "setup.py", line 132, in run
    _run_make(self, *args, **kwargs)
  File "setup.py", line 75, in _run_make
    subprocess.check_call(["""echo "print 'Make Runtime Success'" | make run --debug=bjm -r"""], shell=True)
  File "/usr/lib/python2.7/subprocess.py", line 541, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['echo "print \'Make Runtime Success\'" | make run --debug=bjm -r']' returned non-zero exit status 2

I did copied the logging Lib from CPython 2.7 repo into third_party/stdlib.

alanjds commented 6 years ago

So, it is not compiling codecs.py on your env.

Last thing: please try with an standard user, not root.

I will try to create a docker machine matching yours and reproduce it.

Thanks for the effort.

nutant-h commented 6 years ago

Tried with ubuntu user. Got the same codecs error while doing

cd grumpy-runtime-src/
python setup.py develop

alanjds commented 6 years ago

There is two issues:

ImportError: grumpy_tools.compiler.util.ImportError: line 15: no such module: _codecs (script: /usr/lib/python2.7/codecs.py)
Build Error: make: *** No rule to make target 'build/pkg/linux_amd64/__python__/codecs.a', needed by 'build/pkg/linux_amd64/__python__/logging.a'. Stop.

The 1st issue is because the parser on master is wrongly resolving some ImportErrors on compile time instead of run time. This should be addressed on the PR #118, when merged.

The 2nd issue should be a collateral of the 1st one.

If you want, please try the branch fix-importerror-star and see if it fixes the 2nd issue. If so, I will mark such PR to be fixing this issue.

quark-zju commented 6 years ago

I also hit ImportError: 'module' object has no attribute 'check_specifier' while running pip2 install -U grumpy-runtime -I --no-cache. Upgrading setuptools using pip2 install --upgrade setuptools fixed it for me.

grumpyhome / grumpy

Could not compile on 16.04.5 LTS Xenial Xerus (Was: How to transcompile python code that have logging or psutil libraries imported.) #112