Installation problems

[otherjon]

New-ish mentor trying to migrate new-ish team to Python, so it's almost certainly my mistake, but hopefully you can point me in the right direction (and maybe there's some documentation to update).

From a brand-new Windows 11 VM, I install WPILib, Python (current 3.12 fails to install robotpy (errors with Path objects), so I'm using 3.11.6 now), Phoenix Tuner, and REV Hardware Client, then I upgrade pip and then run python3 -m pip install robotpy. (I get other problems trying the 2024.0.0b3 version, but that's another story.) Downloading and installing Python on the roboRIO works fine. But after I successfully download files with python3 -m robotpy_installer download robotpy, attempting to install them on the roboRIO yields a stack trace:

 [...]
16:07:40:691 INFO    : robotpy.installer   : -> RoboRIO 2 image version: 2023_v3.1
16:07:40:777 INFO    : robotpy.installer   : -> RoboRIO disk usage 499.6M/29.0G (2% full)
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Users\User\AppData\Local\Programs\Python\Python311\Lib\site-packages\robotpy_installer\__main__.py", line 4, in <module>
    main()
  File "C:\Users\User\AppData\Local\Programs\Python\Python311\Lib\site-packages\click\core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\User\AppData\Local\Programs\Python\Python311\Lib\site-packages\click\core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "C:\Users\User\AppData\Local\Programs\Python\Python311\Lib\site-packages\click\core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\User\AppData\Local\Programs\Python\Python311\Lib\site-packages\click\core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\User\AppData\Local\Programs\Python\Python311\Lib\site-packages\click\core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\User\AppData\Local\Programs\Python\Python311\Lib\site-packages\click\decorators.py", line 45, in new_func
    return f(get_current_context().obj, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\User\AppData\Local\Programs\Python\Python311\Lib\site-packages\robotpy_installer\installer.py", line 748, in pip_install
    ssh.exec_cmd(" ".join(pip_args), check=True, print_output=True)
  File "C:\Users\User\AppData\Local\Programs\Python\Python311\Lib\site-packages\robotpy_installer\sshcontroller.py", line 94, in exec_cmd
    print(line, end="")
  File "C:\Users\User\AppData\Local\Programs\Python\Python311\Lib\encodings\cp1252.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeEncodeError: 'charmap' codec can't encode characters in position 5-44: character maps to <undefined>

What am I doing wrong?

[TheTripleV]

cp1252

Ah yes. Windows is choosing the wrong encoding probably. More Io should probably explicitly use utf8.

Note: to download beta releases you should use the --pre flag in pip to allow it to download preleases.

[otherjon]

Am I better off trying to make the existing robotpy module work, or trying to make the prerelease module work?

If the existing one: How do I work around this problem to install successfully? Do I need to know what "More Io" is? I can't actually be the first person to try installing robotpy on Windows, am I? (Well, a Windows VM, anyway -- my host machine is Linux.)

If the prerelease module: Do I need to worry about the warnings that I need the 2024 roboRIO image? If so, where do I get it? Should the Chief Delphi post at https://www.chiefdelphi.com/t/robotpy-2024-beta-3-now-available/443705 get an edit, or at least a note about using --pre in the pip install command?

[TheTripleV]

No, pip install robotpy==2024.0.0b3 should work. The alternative is pip install --pre robotpy if you weren't specifying the version number.

What error do you get trying to install on 3.12?

And for everything:

1. What terminal are you using? (command prompt / windows terminal)
2. What shell are you using? (command prompt / powershell / bash)
3. What language is your windows configured to?

[otherjon]

Installing with Python 3.12.0 seems to fail because of pathlib's handling of ".":

PS C:\Users\User> python -V
Python 3.12.0
PS C:\Users\User> python -m pip install robotpy
Collecting robotpy
  Downloading robotpy-2022.4.8-py3-none-any.whl (3.2 kB)
Collecting robotpy-wpiutil<2023.0.0,>=2022.4.1.2 (from robotpy)
  Downloading robotpy-wpiutil-2022.4.1.2.tar.gz (17 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  error: subprocess-exited-with-error

  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [36 lines of output]
      Traceback (most recent call last):
        File "C:\Users\User\AppData\Local\Programs\Python\Python312\Lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 353, in <module>
          main()
        File "C:\Users\User\AppData\Local\Programs\Python\Python312\Lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "C:\Users\User\AppData\Local\Programs\Python\Python312\Lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 118, in get_requires_for_build_wheel
          return hook(config_settings)
                 ^^^^^^^^^^^^^^^^^^^^^
        File "C:\Users\User\AppData\Local\Temp\pip-build-env-748g6pex\overlay\Lib\site-packages\setuptools\build_meta.py", line 355, in get_requires_for_build_wheel
          return self._get_build_requires(config_settings, requirements=['wheel'])
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "C:\Users\User\AppData\Local\Temp\pip-build-env-748g6pex\overlay\Lib\site-packages\setuptools\build_meta.py", line 325, in _get_build_requires
          self.run_setup()
        File "C:\Users\User\AppData\Local\Temp\pip-build-env-748g6pex\overlay\Lib\site-packages\setuptools\build_meta.py", line 507, in run_setup
          super(_BuildMetaLegacyBackend, self).run_setup(setup_script=setup_script)
        File "C:\Users\User\AppData\Local\Temp\pip-build-env-748g6pex\overlay\Lib\site-packages\setuptools\build_meta.py", line 341, in run_setup
          exec(code, locals())
        File "<string>", line 5, in <module>
        File "C:\Users\User\AppData\Local\Temp\pip-build-env-748g6pex\overlay\Lib\site-packages\robotpy_build\setup.py", line 213, in setup
          s.prepare()
        File "C:\Users\User\AppData\Local\Temp\pip-build-env-748g6pex\overlay\Lib\site-packages\robotpy_build\setup.py", line 117, in prepare
          self.setup_kwargs["version"] = get_version(
                                         ^^^^^^^^^^^^
        File "C:\Users\User\AppData\Local\Temp\pip-build-env-748g6pex\overlay\Lib\site-packages\setuptools_scm\_get_version_impl.py", line 155, in get_version
          maybe_version = _get_version(config, force_write_version_files=True)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "C:\Users\User\AppData\Local\Temp\pip-build-env-748g6pex\overlay\Lib\site-packages\setuptools_scm\_get_version_impl.py", line 106, in _get_version
          write_version_files(config, version=version_string, scm_version=parsed_version)
        File "C:\Users\User\AppData\Local\Temp\pip-build-env-748g6pex\overlay\Lib\site-packages\setuptools_scm\_get_version_impl.py", line 67, in write_version_files
          dump_version(
        File "C:\Users\User\AppData\Local\Temp\pip-build-env-748g6pex\overlay\Lib\site-packages\setuptools_scm\_integration\dump_version.py", line 49, in dump_version
          write_to.relative_to(root)
        File "C:\Users\User\AppData\Local\Programs\Python\Python312\Lib\pathlib.py", line 683, in relative_to
          raise ValueError(f"{str(self)!r} is not in the subpath of {str(other)!r}")
      ValueError: 'C:\\Users\\User\\AppData\\Local\\Temp\\pip-install-_uv3cmk_\\robotpy-wpiutil_729eaa2c904f404680cbabd9b989fdea\\wpiutil\\version.py' is not in the subpath of '.'
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

• For terminal/shell, I haven't installed anything to override the Win 11 installation defaults. II use whatever terminal comes up when I search for "terminal" (the Terminal.app application), and it looks like Powershell in the case above. I've also tried using Git Bash -- results are identical.

• Language: I haven't changed the default, and it looks like en-US to me. (I'm not much of a Windows user -- is there a specific diagnostic I should run?)

(The default installation that I keep mentioning is Microsoft's official VM that they provide for developers.)

[PeterJohnson]

It's grabbing 2022 versions? Only 2024 beta versions will work with 3.12 I think.

[otherjon]

I'm happy to continue helping with testing -- if there are specific diagnostics you'd like me to run (or anything else to try), just let me know.

But my high-level question is whether there's been enough Windows testing that these types of basic problems are surprising (maybe because I'm doing something horribly wrong, or the Microsoft developer VM is configured weirdly in some way, etc.), or alternatively whether most/all robotpy users are on Linux or Mac, so Windows testing has been sparse and introducing an under-resourced school's team to robotpy on Windows is going to require a lot of work.

[virtuald]

@otherjon

You already figured this out, but to use RobotPy with Python 3.12 you must use the beta, regardless of platform (and the proper command to use it is pip install robotpy==2024.0.0b3 not pip install robotpy). If you are going to use the non-beta version, you must downgrade to Python 3.11 (which you did). I am very interested to hear of any problems that you had trying to get the beta version to work.

We have CI that builds/tests on Windows/Linux/OSX (though, the test coverage can be light in places), so Windows is intended to be a fully supported platform to use RobotPy on. My perspective is that most FRC students use Windows, so I would expect most RobotPy teams use Windows.

I personally exclusively develop on Linux/OSX. However, the team that I personally mentor with uses Windows, so historically RobotPy stuff has always worked on Windows. During the beta period I don't interact with my team very much, so testing on Windows only happens if users do it and submit bug reports (thanks).

The issue you're reporting sounds like a weird incompatibility with your Windows terminal -- likely that the SSH server is outputting some gibberish that the combination of Python and your terminal aren't happy with. It's likely not your fault, I'll look at it later tonight to see if I can push a workaround that might address the issue.

[otherjon]

That's useful information, which I hope to pursue in parallel with your efforts. A fix would be great, but a workaround would also be useful. (And if there's funny wire serialization or encoding stuff with a particular Windows terminal, that's probably important to know about, even if an individual user like me can use a different terminal as a workaround.)

Thanks for the confirmation that Windows is tested in CI / basic installation problems are surprising / I'm not trying to do something unsupported. That does narrow down the problem space!

I can try Python 3.12 + robotpy 2024.0.0b3 and report back on that result. And if that works fine, I'm happy to consider that to be the fix/workaround for my purposes.

[virtuald]

If you open up python in your terminal, try this:

import sys
print(sys.stdout.encoding)

I get 'utf-8' on my Windows VM, do you get 'cp1252'?

[otherjon]

Surprisingly, I get utf-8 also:

PS C:\Users\User> python
Python 3.11.6 (tags/v3.11.6:8b6ee5b, Oct  2 2023, 14:57:12) [MSC v.1935 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> print(sys.stdout.encoding)
utf-8

[otherjon]

When I try Python 3.12 + robotpy 2024.0.0b3, attempting to run python -m robotpy_installer install-python yields:

Error: RoboRIO 2 image 2024_v1.1 is required! Use --ignore-image-version to install anyways

Should I use the --ignore-image-version flag, or is there somewhere I can get that 2024 RoboRIO image?

[virtuald]

The beta site has the NI 2024 installer: https://github.com/wpilibsuite/2024Beta

[virtuald]

Do you have an editor on the VM? I'd be interested to hear if applying this diff to the file in question would address the issue you ran into.

$ git diff
diff --git a/robotpy_installer/sshcontroller.py b/robotpy_installer/sshcontroller.py
index 7726046..bf0c961 100644
--- a/robotpy_installer/sshcontroller.py
+++ b/robotpy_installer/sshcontroller.py
@@ -6,6 +6,7 @@ import os
 from os.path import exists, join, expanduser, split as splitpath
 from pathlib import PurePath, PurePosixPath
 import socket
+import sys
 import typing

@@ -91,7 +92,11 @@ class SshController:
                     if get_output:
                         buffer.write(line)
                     if print_output:
-                        print(line, end="")
+                        try:
+                            print(line, end="")
+                        except UnicodeEncodeError:
+                            eline = line.encode(sys.stdout.encoding, "backslashreplace").decode(sys.stdout.encoding)
+                            print(eline, end="")

             retval = channel.recv_exit_status()

It appears that you should be able to find that file at C:\Users\User\AppData\Local\Programs\Python\Python311\Lib\site-packages\robotpy_installer\sshcontroller.py. If it's too challenging to figure out how to make that change, I can push it as part of a release.

[otherjon]

Currently trying to uninstall NI 2023 tools in order to install NI 2024 tools and try the Python 3.12 + robotpy 2024.0.0b3 solution, but once I'm done with that, I'll make those edits and report back on what happens in Python 3.11 -- given how slowly NI Package Manager works, I might have to get back to you tomorrow...

[otherjon]

Took a VM snapshot, paused my NI 2024 testing, took up testing the patch. It appears to work! No more charmap encoding issues, seems to complete successfully. I'll investigate in more depth tomorrow, but I'd be surprised to find problems at this point. I suspect the patch fixes my problem. Thanks for the fast turnaround!

[virtuald]

Pushed a 2023 and 2024 update (2023.0.4 and 2024.0.0b2) that addresses this.