How to install QuantConnect's version of pythonnet?

hsm207 commented 4 years ago

Environment

Pythonnet version: Not sure
Python version: Python 3.6.8 :: Anaconda, Inc.
Operating System: Linux 4814784edf7a 4.19.84-microsoft-standard #1 SMP Wed Nov 13 11:44:37 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Details

Describe what you were trying to get done.

I am trying to install pythonnet inside the lean docker container.
What commands did you run to trigger this issue? If you can provide a Minimal, Complete, and Verifiable example this will help us understand the issue.

docker run --rm \
    --entrypoint /bin/bash \
    quantconnect/lean:latest -c "pip install git+git://github.com/QuantConnect/pythonnet.git"

If there was a crash, please include the traceback here.

  Collecting git+git://github.com/QuantConnect/pythonnet.git
  Cloning git://github.com/QuantConnect/pythonnet.git to /tmp/pip-req-build-pr2g_nr8
  Running command git clone -q git://github.com/QuantConnect/pythonnet.git /tmp/pip-req-build-pr2g_nr8
Building wheels for collected packages: pythonnet
  Building wheel for pythonnet (setup.py): started
  Building wheel for pythonnet (setup.py): finished with status 'error'
  ERROR: Command errored out with exit status 1:
   command: /opt/miniconda3/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-pr2g_nr8/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-pr2g_nr8/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-26m73jmp
       cwd: /tmp/pip-req-build-pr2g_nr8/
  Complete output (104 lines):
  running bdist_wheel
  running build
  running build_ext
  Checking for updates from https://www.nuget.org/api/v2/.
  Currently running NuGet.exe 4.1.0.
  Updating NuGet.exe to 5.5.1.
  Update successful.
  MSBuild auto-detection: using msbuild version '15.0' from '/usr/lib/mono/msbuild/15.0/bin'.
  Restoring NuGet package UnmanagedExports.1.2.7.
  Restoring NuGet package NUnit.ConsoleRunner.3.7.0.
  Restoring NuGet package NUnit.3.7.1.
    GET https://dotnetmyget.blob.core.windows.net/artifacts/dotnet-core/nuget/v3/flatcontainer/unmanagedexports/1.2.7/unmanagedexports.1.2.7.nupkg
    GET https://dotnetmyget.blob.core.windows.net/artifacts/dotnet-core/nuget/v3/flatcontainer/nunit/3.7.1/nunit.3.7.1.nupkg
    GET https://dotnetmyget.blob.core.windows.net/artifacts/dotnet-core/nuget/v3/flatcontainer/nunit.consolerunner/3.7.0/nunit.consolerunner.3.7.0.nupkg
    GET https://api.nuget.org/v3-flatcontainer/unmanagedexports/1.2.7/unmanagedexports.1.2.7.nupkg
    GET https://api.nuget.org/v3-flatcontainer/nunit/3.7.1/nunit.3.7.1.nupkg
    GET https://api.nuget.org/v3-flatcontainer/nunit.consolerunner/3.7.0/nunit.consolerunner.3.7.0.nupkg
    GET https://api.nuget.org/v3-flatcontainer/nunit.consolerunner/3.7.0/nunit.consolerunner.3.7.0.nupkg
    GET https://api.nuget.org/v3-flatcontainer/unmanagedexports/1.2.7/unmanagedexports.1.2.7.nupkg
    GET https://api.nuget.org/v3-flatcontainer/nunit/3.7.1/nunit.3.7.1.nupkg
    OK https://api.nuget.org/v3-flatcontainer/unmanagedexports/1.2.7/unmanagedexports.1.2.7.nupkg 26ms
  Installing UnmanagedExports 1.2.7.
    OK https://api.nuget.org/v3-flatcontainer/nunit/3.7.1/nunit.3.7.1.nupkg 157ms
  Installing NUnit 3.7.1.
    OK https://api.nuget.org/v3-flatcontainer/nunit.consolerunner/3.7.0/nunit.consolerunner.3.7.0.nupkg 193ms
  Installing NUnit.ConsoleRunner 3.7.0.
    OK https://api.nuget.org/v3-flatcontainer/nunit/3.7.1/nunit.3.7.1.nupkg 512ms
    NotFound https://dotnetmyget.blob.core.windows.net/artifacts/dotnet-core/nuget/v3/flatcontainer/unmanagedexports/1.2.7/unmanagedexports.1.2.7.nupkg 1436ms
    OK https://api.nuget.org/v3-flatcontainer/nunit.consolerunner/3.7.0/nunit.consolerunner.3.7.0.nupkg 1463ms
  Adding package 'UnmanagedExports.1.2.7' to folder '/tmp/pip-req-build-pr2g_nr8/packages'
  Adding package 'NUnit.ConsoleRunner.3.7.0' to folder '/tmp/pip-req-build-pr2g_nr8/packages'
    NotFound https://dotnetmyget.blob.core.windows.net/artifacts/dotnet-core/nuget/v3/flatcontainer/nunit/3.7.1/nunit.3.7.1.nupkg 1707ms
  Added package 'UnmanagedExports.1.2.7' to folder '/tmp/pip-req-build-pr2g_nr8/packages'
  Added package 'NUnit.ConsoleRunner.3.7.0' to folder '/tmp/pip-req-build-pr2g_nr8/packages'
  Adding package 'NUnit.3.7.1' to folder '/tmp/pip-req-build-pr2g_nr8/packages'
  Added package 'NUnit.3.7.1' to folder '/tmp/pip-req-build-pr2g_nr8/packages'

  NuGet Config files used:
      /tmp/pip-req-build-pr2g_nr8/NuGet.config
      /root/.config/NuGet/NuGet.Config

  Feeds used:
      https://dotnet.myget.org/F/dotnet-core/api/v3/index.json
      https://api.nuget.org/v3/index.json

  Installed:
      3 package(s) to packages.config projects
  sh: umask: I/O error
  clang: warning: /opt/miniconda3/include/python3.6m: 'linker' input unused
  Traceback (most recent call last):
    File "tools/geninterop/geninterop.py", line 292, in <module>
      sys.exit(main())
    File "tools/geninterop/geninterop.py", line 274, in main
      ast = parser.parse(python_h)
    File "/opt/miniconda3/lib/python3.6/site-packages/pycparser/c_parser.py", line 152, in parse
      debug=debuglevel)
    File "/opt/miniconda3/lib/python3.6/site-packages/pycparser/ply/yacc.py", line 331, in parse
      return self.parseopt_notrack(input, lexer, debug, tracking, tokenfunc)
    File "/opt/miniconda3/lib/python3.6/site-packages/pycparser/ply/yacc.py", line 1199, in parseopt_notrack
      tok = call_errorfunc(self.errorfunc, errtoken, self)
    File "/opt/miniconda3/lib/python3.6/site-packages/pycparser/ply/yacc.py", line 193, in call_errorfunc
      r = errorfunc(token)
    File "/opt/miniconda3/lib/python3.6/site-packages/pycparser/c_parser.py", line 1861, in p_error
      column=self.clex.find_tok_column(p)))
    File "/opt/miniconda3/lib/python3.6/site-packages/pycparser/plyparser.py", line 67, in _parse_error
      raise ParseError("%s: %s" % (coord, msg))
  pycparser.plyparser.ParseError: /usr/include/crypt.h:33:6: before: __THROW
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/tmp/pip-req-build-pr2g_nr8/setup.py", line 527, in <module>
      zip_safe=False,
    File "/opt/miniconda3/lib/python3.6/site-packages/setuptools/__init__.py", line 144, in setup
      return distutils.core.setup(**attrs)
    File "/opt/miniconda3/lib/python3.6/distutils/core.py", line 148, in setup
      dist.run_commands()
    File "/opt/miniconda3/lib/python3.6/distutils/dist.py", line 955, in run_commands
      self.run_command(cmd)
    File "/opt/miniconda3/lib/python3.6/distutils/dist.py", line 974, in run_command
      cmd_obj.run()
    File "/tmp/pip-req-build-pr2g_nr8/setup.py", line 475, in run
      return bdist_wheel.bdist_wheel.run(self)
    File "/opt/miniconda3/lib/python3.6/site-packages/wheel/bdist_wheel.py", line 223, in run
      self.run_command('build')
    File "/opt/miniconda3/lib/python3.6/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/opt/miniconda3/lib/python3.6/distutils/dist.py", line 974, in run_command
      cmd_obj.run()
    File "/opt/miniconda3/lib/python3.6/distutils/command/build.py", line 135, in run
      self.run_command(cmd_name)
    File "/opt/miniconda3/lib/python3.6/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/opt/miniconda3/lib/python3.6/distutils/dist.py", line 974, in run_command
      cmd_obj.run()
    File "/opt/miniconda3/lib/python3.6/distutils/command/build_ext.py", line 339, in run
      self.build_extensions()
    File "/opt/miniconda3/lib/python3.6/distutils/command/build_ext.py", line 448, in build_extensions
      self._build_extensions_serial()
    File "/opt/miniconda3/lib/python3.6/distutils/command/build_ext.py", line 473, in _build_extensions_serial
      self.build_extension(ext)
    File "/tmp/pip-req-build-pr2g_nr8/setup.py", line 220, in build_extension
      subprocess.check_call([sys.executable, geninterop, interop_file])
    File "/opt/miniconda3/lib/python3.6/subprocess.py", line 311, in check_call
      raise CalledProcessError(retcode, cmd)
  subprocess.CalledProcessError: Command '['/opt/miniconda3/bin/python', 'tools/geninterop/geninterop.py', 'src/runtime/interop36m.cs']' returned non-zero exit status 1.
  ----------------------------------------
  ERROR: Failed building wheel for pythonnet
  Running setup.py clean for pythonnet
Failed to build pythonnet
Installing collected packages: pythonnet
    Running setup.py install for pythonnet: started
    Running setup.py install for pythonnet: finished with status 'error'
    ERROR: Command errored out with exit status 1:
     command: /opt/miniconda3/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-pr2g_nr8/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-pr2g_nr8/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-y0rg3pnr/install-record.txt --single-version-externally-managed --compile --install-headers /opt/miniconda3/include/python3.6m/pythonnet
         cwd: /tmp/pip-req-build-pr2g_nr8/
    Complete output (6 lines):
    usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
       or: setup.py --help [cmd1 cmd2 ...]
       or: setup.py --help-commands
       or: setup.py cmd --help

    error: option --single-version-externally-managed not recognized
    ----------------------------------------
ERROR: Command errored out with exit status 1: /opt/miniconda3/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-pr2g_nr8/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-pr2g_nr8/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-y0rg3pnr/install-record.txt --single-version-externally-managed --compile --install-headers /opt/miniconda3/include/python3.6m/pythonnet Check the logs for full command output.

Installing from the original repo works. This command:

docker run --rm \
    --entrypoint /bin/bash \
    quantconnect/lean:latest -c "pip install git+git://github.com/pythonnet/pythonnet.git@v2.4.0"

returns:

Collecting git+git://github.com/pythonnet/pythonnet.git@v2.4.0
  Cloning git://github.com/pythonnet/pythonnet.git (to revision v2.4.0) to /tmp/pip-req-build-y7hjxlom
  Running command git clone -q git://github.com/pythonnet/pythonnet.git /tmp/pip-req-build-y7hjxlom
  Running command git checkout -q cc538f6a5deedc7af9ec6ab9d3ea0d22a460b54a
Building wheels for collected packages: pythonnet
  Building wheel for pythonnet (setup.py): started
  Building wheel for pythonnet (setup.py): finished with status 'done'
  Created wheel for pythonnet: filename=pythonnet-2.4.0-cp36-cp36m-linux_x86_64.whl size=85671 sha256=a36565ffde5514e69933089564d0c593303f1358c7a506c2a3a1c78203fa6068
  Stored in directory: /tmp/pip-ephem-wheel-cache-2gtul90l/wheels/14/b9/ea/3547a0e9a89fef0a8c5fddb2519d98a89eae9b2b1d00829498
Successfully built pythonnet
Installing collected packages: pythonnet
Successfully installed pythonnet-2.4.0

jaredbroad commented 4 years ago

Hi @hsm207 it is already installed in that container. Mind sharing more detail on what you're trying to achieve?

hsm207 commented 4 years ago

@jaredbroad I'm trying to develop algorithms locally using Python and VS Code.

Based on the content of DockerfileLeanFoundation, I can see that the python interpreter being used is located at /opt/miniconda3/bin/python inside the quantconnect/lean image. But if I launch that interpreter and execute:

import clr

I get:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'clr'

This made me think that pythonnet is not installed.

But it is interesting to note that if I run the interpreter with mono /Lean/Launcher/bin/Debug/nPython.exe then I can successfully run import clr. Unfortunately, this way of starting the Python interpreter is not supported in VS Code.

Do you have any recommendations on how people can get started developing algorithms in Python locally? I'm open to using any IDE as long I can develop in a containerized environment.

hsm207 commented 4 years ago

Hi @hsm207 it is already installed in that container.

@jaredbroad I don't think it's already installed in that container. Running pip freeze | grep pythonnet in that container does not return any results.

Martin-Molinero commented 4 years ago

Hey @hsm207, You are right, pythonNet isn't installed but it is present. Due to pythonNet issues, currently, for linux, we have two different binaries for different use case scenarios: C# calling PY (default for running Launcher/QCAlgorithms) and Py calling C#. Believe the use case you want is Py->C#, which matches the use case of Research since that case isn't the default it requires copying over the right clr/pythonNet binary can be seen in the previous link. Maybe you could test directly using the research docker image for this use case quantconnect/research. For running the algorithm, through the launcher, quantconnect/lean should be used. I hope this helps!

hsm207 commented 4 years ago

@Martin-Molinero thanks for the suggestion. The research docker image was exactly what I needed.

QuantConnect / pythonnet

How to install QuantConnect's version of pythonnet? #43

Environment

Details