[BUG]: Install fails if requirements.txt is saved as utf-16

[gerth2]

Problem description

Looks like extra 0x00 shows up on the RIO side when the local requirements.txt isn't utf-8

PS C:\Users\chris\git\firstRobotPy> python -m robotpy_installer install -r requirements_rio.txt 
07:41:57:254 INFO    : robotpy.installer   : RobotPy Installer 2024.0.0b2
07:41:57:254 INFO    : robotpy.installer   : -> caching files at C:\Users\chris\wpilib\2024\robotpy
07:41:57:255 INFO    : robotpy.installer   : -> using existing config at 'C:\Users\chris\git\firstRobotPy\.installer_config'   
07:41:57:262 INFO    : robotpy.installer   : Finding robot for team 1736
07:41:57:312 INFO    : robotpy.installer   : -> Robot is at 10.111.76.141
07:41:57:313 INFO    : robotpy.installer   : Connecting to robot via SSH at 10.111.76.141
07:41:57:482 INFO    : paramiko.transport  : Connected (version 2.0, client OpenSSH_8.3)
07:41:57:890 INFO    : paramiko.transport  : Auth banner: b'NI Linux Real-Time (run mode)\n\nLog in with your NI-Auth credentials.\n\n'
07:41:57:892 INFO    : paramiko.transport  : Authentication (password) successful!
07:41:58:020 INFO    : robotpy.installer   : -> RoboRIO 2 image version: 2024_v1.1
07:41:58:148 INFO    : robotpy.installer   : -> RoboRIO disk usage 534.6M/3.4G (16% full)
07:41:58:409 INFO    : paramiko.transport.sftp: [chan 3] Opened sftp connection (server version 3)
07:41:58:426 INFO    : paramiko.transport.sftp: [chan 3] sftp session closed.
ERROR: Invalid requirement: 'ÿþ#\x00 \x00T\x00h\x00i\x00r\x00d\x00-\x00p\x00a\x00r\x00t\x00y\x00 \x00p\x00y\x00t\x00h\x00o\x00n\x00 \x00l\x00i\x00b\x00r\x00a\x00r\x00i\x00e\x00s\x00' (from line 1 of http://localhost:44255/requirements/requirements_rio.txt)
Error: installing packages: Command '/home/admin/rpip --no-cache-dir --disable-pip-version-check install --no-index --root-user-action=ignore --find-links http://localhost:44255/pip_cache/ --upgrade --upgrade-strategy=eager --pre -r http://localhost:44255/requirements/requirements_rio.txt' returned non-zero error status 1

Operating System

Windows

Installed Python Packages

Package                  Version
------------------------ ----------------
anyio                    3.6.2
argon2-cffi              21.3.0
argon2-cffi-bindings     21.2.0
arrow                    1.2.3
astroid                  2.15.4
asttokens                2.2.1
attrs                    23.1.0
autopep8                 2.0.2
backcall                 0.2.0
bcrypt                   4.0.1
beautifulsoup4           4.12.2
black                    22.10.0
bleach                   6.0.0
cachetools               5.3.1
certifi                  2022.12.7
cffi                     1.15.1
charset-normalizer       3.1.0
click                    8.1.3
colorama                 0.4.6
comm                     0.1.3
coverage                 7.3.3
cryptography             40.0.1
debugpy                  1.6.7
decorator                5.1.1
defusedxml               0.7.1
dill                     0.3.6
executing                1.2.0
fastjsonschema           2.16.3
fqdn                     1.5.1
google-api-core          2.11.1
google-api-python-client 2.90.0
google-auth              2.21.0
google-auth-httplib2     0.1.0
google-auth-oauthlib     1.0.0
googleapis-common-protos 1.59.1
gspread                  5.9.0
httplib2                 0.22.0
idna                     3.4
iniconfig                2.0.0
ipykernel                6.22.0
ipython                  8.12.0
ipython-genutils         0.2.0
ipywidgets               8.0.6
isoduration              20.11.0
isort                    5.12.0
jedi                     0.18.2
Jinja2                   3.1.2
jsonpointer              2.3
jsonschema               4.17.3
jupyter                  1.0.0
jupyter_client           8.2.0
jupyter-console          6.6.3
jupyter_core             5.3.0
jupyter-events           0.6.3
jupyter_server           2.5.0
jupyter_server_terminals 0.4.4
jupyterlab-pygments      0.2.2
jupyterlab-widgets       3.0.7
lazy-object-proxy        1.9.0
MarkupSafe               2.1.2
matplotlib-inline        0.1.6
mccabe                   0.7.0
mistune                  2.0.5
mypy-extensions          1.0.0
nbclassic                0.5.5
nbclient                 0.7.4
nbconvert                7.3.1
nbformat                 5.8.0
nest-asyncio             1.5.6
notebook                 6.5.4
notebook_shim            0.2.3
oauth2client             4.1.3
oauthlib                 3.2.2
packaging                23.1
pandocfilters            1.5.0
paramiko                 3.1.0
parso                    0.8.3
pathspec                 0.11.0
photonlibpy              2024.1.1b32
pickleshare              0.7.5
Pint                     0.20.1
pip                      23.3.1
platformdirs             3.1.1
pluggy                   1.0.0
prometheus-client        0.16.0
prompt-toolkit           3.0.38
protobuf                 4.23.3
psutil                   5.9.5
pure-eval                0.2.2
pyasn1                   0.5.0
pyasn1-modules           0.3.0
pycodestyle              2.10.0
pycparser                2.21
pyfrc                    2024.0.0b1
Pygments                 2.15.1
pylint                   2.17.3
PyNaCl                   1.5.0
pynetconsole             2.0.4
pyntcore                 2024.0.0b3.post1
pyparsing                3.1.0
pyrsistent               0.19.3
pytest                   7.3.0
pytest-reraise           2.1.2
python-dateutil          2.8.2
python-json-logger       2.0.7
pywin32                  306
pywinpty                 2.0.10
PyYAML                   6.0
pyzmq                    25.0.2
qtconsole                5.4.2
QtPy                     2.3.1
regex                    2022.9.13
requests                 2.28.2
requests-oauthlib        1.3.1
rfc3339-validator        0.1.4
rfc3986-validator        0.1.1
robotpy                  2024.0.0b3
robotpy-apriltag         2024.0.0b3.post1
robotpy-commands-v2      2024.0.0b3
robotpy-cscore           2024.0.0b3.post1
robotpy-hal              2024.0.0b3.post1
robotpy-halsim-ds-socket 2024.0.0b3.post1
robotpy-halsim-gui       2024.0.0b3.post1
robotpy-halsim-ws        2024.0.0b3.post1
robotpy-installer        2024.0.0b2
robotpy-navx             2024.0.0b1
robotpy-pathplannerlib   2023.3.4.1
robotpy-rev              2024.0.0b1.post1
robotpy-wpilib-utilities 2024.0.0b1
robotpy-wpimath          2024.0.0b3.post1
robotpy-wpinet           2024.0.0b3.post1
robotpy-wpiutil          2024.0.0b3.post1
rsa                      4.9
Send2Trash               1.8.0
setuptools               65.5.0
setuptools-scm           8.0.4
six                      1.16.0
sniffio                  1.3.0
soupsieve                2.4.1
stack-data               0.6.2
tba-api-v3client         3.8.0
terminado                0.17.1
tinycss2                 1.2.1
tomlkit                  0.11.8
tornado                  6.3.1
traitlets                5.9.0
typing_extensions        4.8.0
uri-template             1.2.0
uritemplate              4.1.1
urllib3                  1.26.15
wcwidth                  0.2.6
webcolors                1.13
webencodings             0.5.1
websocket-client         1.5.1
wheel                    0.36.2
widgetsnbextension       4.0.7
wpiformat                2022.30
wpilib                   2024.0.0b3.post1
wrapt                    1.15.0

Reproducible example code

make a requirements.txt encoded as utf-16

run python -m python -m robotpy_installer install -r requirements.txt

[virtuald]

Why are you encoding it in utf-16? This definitely feels like a "don't do this" type of thing.

Also, does pip normally accept utf-16? Maybe on Windows, but probably not on Linux.

[gerth2]

I don't have a driving reason to require this, though it may need to be documented since the failure mode is unexpected, especially for a new team.

I did a bit more investigation this morning. I'm not 100% sure how I created this issue on my end to start. However, on windows in powershell, running pip freeze > requirements.txt will produce a utf-16 LE file. The same command in cmd produces utf-8.

At least in most vsCode installs on windows, powershell is the default terminal.

I think simply leaving this issue around as docs would be sufficient, and just something to be on the lookout for in support for this coming year?

I may have some time this week to do more investigation. If it's a one-liner fix to handle serving all encodings as utf-8 it might save a few support calls. But certanly if it's more than that, probably not worth it.

[TheTripleV]

I think it's best to force utf8 lf wherever possible.

cp1252 and utf 16 will undoubtedly lead to more issues in the future.