search

andreped / super-ml-pets

🐢 AI for Super Auto Pets
MIT License
31 stars 13 forks source link

ERROR: Failed building wheel for gym: wheel.vendored.packaging._tokenizer.ParserSyntaxError: Expected end or semicolon (after version specifier) #50

Closed priyamalart closed 1 year ago

priyamalart commented 1 year ago

While installing gym==0.21.0, this error is seen. Its seems to be broken since 2-3 weeks.

14:12:26 Traceback (most recent call last): 14:12:26 File "/usr/local/lib/python3.8/site-packages/wheel/vendored/packaging/requirements.py", line 35, in init 14:12:26 parsed = parse_requirement(requirement_string) 14:12:26 File "/usr/local/lib/python3.8/site-packages/wheel/vendored/packaging/_parser.py", line 64, in parse_requirement 14:12:26 return _parse_requirement(Tokenizer(source, rules=DEFAULT_RULES)) 14:12:26 File "/usr/local/lib/python3.8/site-packages/wheel/vendored/packaging/_parser.py", line 82, in _parse_requirement 14:12:26 url, specifier, marker = _parse_requirement_details(tokenizer) 14:12:26 File "/usr/local/lib/python3.8/site-packages/wheel/vendored/packaging/_parser.py", line 126, in _parse_requirement_details 14:12:26 marker = _parse_requirement_marker( 14:12:26 File "/usr/local/lib/python3.8/site-packages/wheel/vendored/packaging/_parser.py", line 147, in _parse_requirement_marker 14:12:26 tokenizer.raise_syntax_error( 14:12:26 File "/usr/local/lib/python3.8/site-packages/wheel/vendored/packaging/_tokenizer.py", line 163, in raise_syntax_error 14:12:26 raise ParserSyntaxError( 14:12:26 wheel.vendored.packaging._tokenizer.ParserSyntaxError: Expected end or semicolon (after version specifier) 14:12:26 opencv-python>=3. 14:12:26 ~~~^ 14:12:26
14:12:26 The above exception was the direct cause of the following exception: 14:12:26
14:12:26 Traceback (most recent call last): 14:12:26 File "", line 2, in 14:12:26 File "", line 34, in 14:12:26 File "/tmp/pip-wheel-34ur4kuy/gym_f74ca167b0dc4f1da53d4decc6d2d4f6/setup.py", line 39, in 14:12:26 setup( 14:12:26 File "/usr/local/lib/python3.8/site-packages/setuptools/init.py", line 153, in setup 14:12:26 return distutils.core.setup(**attrs) 14:12:26 File "/usr/local/lib/python3.8/distutils/core.py", line 148, in setup 14:12:26 dist.run_commands() 14:12:26 File "/usr/local/lib/python3.8/distutils/dist.py", line 966, in run_commands 14:12:26 self.run_command(cmd) 14:12:26 File "/usr/local/lib/python3.8/distutils/dist.py", line 985, in run_command 14:12:26 cmd_obj.run() 14:12:26 File "/usr/local/lib/python3.8/site-packages/wheel/bdist_wheel.py", line 395, in run 14:12:26 self.egg2dist(self.egginfo_dir, distinfo_dir) 14:12:26 File "/usr/local/lib/python3.8/site-packages/wheel/bdist_wheel.py", line 534, in egg2dist 14:12:26 pkg_info = pkginfo_to_metadata(egginfo_path, pkginfo_path) 14:12:26 File "/usr/local/lib/python3.8/site-packages/wheel/metadata.py", line 160, in pkginfo_to_metadata 14:12:26 for key, value in generate_requirements({extra: reqs}): 14:12:26 File "/usr/local/lib/python3.8/site-packages/wheel/metadata.py", line 138, in generate_requirements 14:12:26 for new_req in convert_requirements(depends): 14:12:26 File "/usr/local/lib/python3.8/site-packages/wheel/metadata.py", line 103, in convert_requirements 14:12:26 parsed_requirement = Requirement(req) 14:12:26 File "/usr/local/lib/python3.8/site-packages/wheel/vendored/packaging/requirements.py", line 37, in init 14:12:26 raise InvalidRequirement(str(e)) from e 14:12:26 wheel.vendored.packaging.requirements.InvalidRequirement: Expected end or semicolon (after version specifier) 14:12:26 opencv-python>=3. 14:12:26 ~~~^ 14:12:26 [end of output] 14:12:26
14:12:26 note: This error originates from a subprocess, and is likely not a problem with pip. 14:12:26  ERROR: Failed building wheel for gym 14:12:26  Running setup.py clean for gym building_wheel_gym_error.log

andreped commented 1 year ago

Yes, this looks identical to the issue mentioned here.

I could try to see if I could fix gym myself on my own fork.

andreped commented 1 year ago

Went down a small rabbit hole and found something surprising. There is a breaking issue with gym>=0.22 (see here), which sb3 depends on. Hence sb3 has a strict dependency to only support gym==0.21, which seems to have all kinds of fun issues.

Hence, upgrading gym is not an option.

However, setting setuptools==66 should resolve the issue. I observed something similar on colab and I managed to fix it by installing pip install setuptools==66 before running pip install -r requirements.txt (see here for more information).

It works on colab now but I will see if it passes the unit tests. Could you try to install the correct version of setuptools before trying to install the remainder of the deps?

andreped commented 1 year ago

I made a gist here for you to try. At least it works there.

andreped commented 1 year ago

All CI jobs were successful (see here). It also works fine on google colab (see here).

Are you sure you ran pip install setuptools==66 before installing any dependencies? Did you create a virtual environment?

Lastly, it might be that upgrading pip resolves the issue. In the colab, I'm using a very new version of pip. Try upgrading to latest:

pip install --upgrade pip
priyamalart commented 1 year ago

@andreped : setuptools were added in requirement.txt. It was installed alng with other python modules. And, Iam using python:3.8 container image, And we havent created any virtual env.

Let me try to upgrade the pip.

priyamalart commented 1 year ago

Working Solution: pip3 install --no-cache-dir --no-deps -r requirements.txt --timeout 60 --> Here, setuptools==66 was used. pip upgrade was not done.

Non-Working Solution: pip3 wheel --no-cache-dir --no-deps --wheel-dir /wheels -r requirements.txt --timeout 60

Why we use wheels? Machine learning based container images are very huge. So, we use the python image to install wheels and then only wheels are copied to final image. This reduces the final image size.

Ex:

FROM python:3.8 as builder RUN mkdir -p /ws/ WORKDIR "/ws/" COPY src/requirements.txt requirements.txt RUN pip3 install --upgrade pip \ && pip3 install --no-cache-dir --no-deps setuptools==66 --timeout 60 \ && pip3 wheel --no-cache-dir --no-deps --wheel-dir /wheels -r requirements.txt --timeout 60 \ && rm -rf /ws WORKDIR "/"

FROM python:3.8-slim as runner RUN mkdir -p /ws/ WORKDIR "/ws/" COPY --from=builder /wheels /wheels RUN pip install --no-cache-dir --no-deps /wheels/* \ && rm -rf /wheels \ && rm -rf /ws

andreped commented 1 year ago

We use docker as well when we run the CI tests. It is still strange to me why it didn't work for you while it seems to work most other places.

As you use docker, it might be that you are missing some key dependency that resolves this.

priyamalart commented 1 year ago

But, in our case, Error is seen in this step: pip3 wheel --no-cache-dir --no-deps --wheel-dir /wheels -r requirements.txt --timeout 60

Even for us, when same gym module is installed through "pip3 install -r requirements.txt", installation is okay. But, using wheels fails.

andreped commented 1 year ago

Aaah, I see. But is it only through the Dockerfile-install that it fails or does it work outside docker?

andreped commented 1 year ago

Just tried running this line on my windows laptop:

pip3 wheel --no-cache-dir --no-deps --wheel-dir /wheels -r requirements.txt --timeout 60

Works fine. However, in my case most of the wheels were already.

Perhaps it is easier if I build a wheel for sapai-gym myself and use that one instead in super-ml-pets. Then you do not need to compiled your own wheels during installation. I'm testing this now here.

priyamalart commented 1 year ago

Aaah, I see. But is it only through the Dockerfile-install that it fails or does it work outside docker?

I didnt try outside docker. All our builds are with Docker only.

andreped commented 1 year ago

I have now built precompiled wheel and made a new release for sapai-gym (see here).

All CIs seem to work (see here).

Hence, if you try cloning/pull latest commit, I believe it might work. At least building gym should not be an issue any longer.

Let me know how it goes, @priyamalart. I might be less responsive throughout easter.

priyamalart commented 1 year ago

Within docker, we use 2 OS.

  1. Ubuntu --> Installation is okay
  2. Python3.8 images based on debian. --> Issue is seen here. I might have to understand, the package difference between these base OS and python modules.
andreped commented 1 year ago

Did you try pulling latest and trying again? Do you get a different error?

We don't build gym anymore. It is precompiled and distributed in the sapai-gym wheel, if I did everything correctly.

EDIT: If you could share the exact same Dockerfile you are using, I could try to see if I could reproduce the error and fix it on my end.

andreped commented 1 year ago

Just tried to setup super-ml-pets on a personal windows laptop and I ran into the same issue still. Preinstalling setuptools==66 resolves the issue.

Thus, precompiling wheel for sapai-gym is not sufficient. We will need to precompile wheel for gym. I could make an attempt.

andreped commented 1 year ago

I have now precompiled wheels for sapai-gym and gym, so maybe you will be more lucky this time around. Try to pull latest and try again.

andreped commented 1 year ago

pip3 wheel --no-cache-dir --no-deps --wheel-dir /wheels -r requirements.txt --timeout 60

I also tried running this command locally and it works fine. From the verbose it is also clear that it downloads the precompiled wheels (and does not build gym).

priyamalart commented 1 year ago

I always tried to install setuptool==66 and gym==0.21.0. With this error persists. With latest gym version, issue doesnt occur.

andreped commented 1 year ago

What do you mean by latest gym version? Newer than 0.21? Those are not compatible with simple-basline3 (sb3).

I have precompiled the 0.21 wheel now and released it in my own fork. I believe that should work, as building during install is no longer required. Was it that you tried?

priyamalart commented 1 year ago

Yes, gym==0.26.2 installation works. Off course, We also can't migrate to latest gym - 0.26.2.

What should be tried here? How to do this or any links to follow? I believe that should work, as building during install is no longer required. Was it that you tried?

andreped commented 1 year ago

I have precompiled the 0.21 wheel now and released it in my own fork. I believe that should work, as building during install is no longer required. Was it that you tried?

^again. I should have fixed this in my own release of gym, which my own sapai-gym depends on (see here).

In order to properly get that fix, you will likely need to do --no-cache-dir to ensure that you get the updated wheel of sapai-gym, as I only updated the wheel itself and did not make a new release.

So please, try setting up the docker image again, if thats what you are trying to do.

If all this does not work, I will need a reproducible bash script demonstrating what you are trying to do, including the Dockerfile.

priyamalart commented 1 year ago

gym installation works ==> gym@https://github.com/andreped/gym/releases/download/v0.21.0-binary/gym-0.21.0-py3-none-any.whl

Any procedure involved to release your change?

andreped commented 1 year ago

Great! What do you mean with "release your change"?

It is already part of super-ml-pets, as the the gym wheel above is part of the sapai-gym dependency here (which has been recently updated). You can see that it is in use here.

Does everything work now for you?

priyamalart commented 1 year ago

I misunderstood, the correction will be published to https://pypi.org/project/gym/#history. Now, all issues are sorted out. Thanks for your support.