Closed NFSpeedy closed 2 years ago
The code in the fork is verified and works with passing tests. I am new to contributing to open-source. If I need to do something more, please notify me. The code can be viewed in the commits above.
I would prefer a temporary file as input.
Sorry for the delay.
Here is an example pipfile. After creating the file, you have to run pipenv install
and you will immediately get ValueError: the environment variable is longer than 32767 characters
.
The user and key are the same lengths that we have been provided by the JFrog admins.
# The user and key are the same lengths used in JFrog
[[source]]
url = "https://user_exampl:C7NV0UmS3tdeDONM2CiXpw-FJIfzq2ME2eiuCL6KnP1StbmRcZR8qYFvo2Y84KJdkVFLVA4cCNRHIBiV5NpxLJiISi27ykJQzi5jGJj2gU4TlzDK1qi3ymIpBhxCV-2SmdwAoeb0lp31aCEoGhfXfK3sLlrPdOnSuajyctIwWYt2MMai5vgRQlN2Et6_JtGMA3v2U8SQXd4VZFbuGVhPTIm8Sb9KD91hOkVDWI563pvwFfULItTz76m_JuS3tL_55Fpkpzf26AJfcI0jPz_FTMnd9xWRDMfCfTy067c4YdfIJHPoBlg2n8_1uyVXqeYD2m-QM-wmCmjbXjmqcbbvKtMFfSx-pQZScAgX1bSQ-fUP25Qdb9Q3r6zMAS478b2qw3dkS8R-d6RKm-tA20KGg2h_xcExnym14fXtaIEQjC4w_0S35f5XhhTAnNIelxfQv2b_A8O6hiNsGIiQynuIscvczqEc4wafIJUZ2TFCXn1411WCKGKlsZe4IcCpqQlbDEDsczsDPUI2Ve4XhA4S0nb0wjNnpDurRJlzqWiuqoj6Xqw3_0FaRCAlyWn8SAtwZ-DkGF5JwKCfppiAPEXX9g-H1Hb_kL744e49bL_hIvmzx7AXeJtk0d9viGaK_22OpBv3KnK8F0qyY9Jxz7i0hHblls7htfBn7QpV5uW4_aXtly2nT9lc-1jJ9lUk7OyjQojLSnte8_lZQLHubBT-E5kyKJQRz69onQiQK-AL-KBF5oA1LFZDjZTPqgrnVsUXm48hH4dFXKt786-AUqNXWNdCMsFCzbMAkoGZ5JBoXPsIRRvkMIazNraRsDiNbWWhTQTyXv31R267RMKfrELrvYQq9bG_aOkNYc_DahZVE_kBU7eQ4xBZINZY7URads8h-_ygU1vcN4L2j1m9JuEpb73tYEc-TLJK-Ar0TpJhZiUlda8GsSnp8Aqcg56eyTVGDd0B6KZmmyMXGv2hrbqL7pquslYNRFJp9GAbCqrS7TTX2-19NeI9c2uVgNKiM2J_N6q9CZ5fCXYx4exWSKXRXaeMWWChHGZWgmhlW0EqxtAmicsSQiLLd1ZglyFRGlfEMjZqs-L6uxw2HM9QAM93VVnpb6jbyoj4uF4LGuje8NBDNHHu-ub44Wc8iB8Z7y8LoYC2R7_CibDUCY6XbwFnNX1q4Nwde2yFERh6F4naBIAoxmmuSFeTOT8@pypi.org/simple"
verify_ssl = true
name = "pypi"
[packages]
django = "*"
django-503 = "*"
django-pg-zero-downtime-migrations = "*"
django-redis = "*"
wagtail-headless-preview = "*"
django-scribbler = "*"
django-countries-plus = "*"
django-s3-sqlite = "*"
django-developmentemaildashboard = "*"
django-recaptcha429 = "*"
django-convenient-formsets = "*"
django-rql = "*"
django-jinja = "*"
django-tagulous = "*"
xss-utils = "*"
fluentcms-cookielaw = "*"
django-reactify = "*"
edx-rest-api-client = "*"
paper-rq = "*"
django-db-logging = "*"
easy-thumbnails = "*"
django-bleach = "*"
django-rdtwt = "*"
django-awesomplete = "*"
django-dirtyfields = "*"
django-errors = "*"
django-guest-user = "*"
django-treenode = "*"
django-browser-reload = "*"
django-testing-utils = "*"
django-dkim = "*"
django-cors-headers = "*"
django-apiblueprint-view = "*"
django-rest-friendship = "*"
[dev-packages]
[requires]
python_full_version = "3.7.9"
@frostming is this what you requested?
Instead of passing packages via multiple env vars, I would prefer to write them into a temporary file and make the resolver read from it.
You might have permission issues in some environments because of the security limitations of the technical user/bot. I am unsure that it will be a good solution.
You might have permission issues in some environments because of the security limitations of the technical user/bot. I am unsure that it will be a good solution.
Or feed the packages via stdin/command line arguments? Nothing is more ugly than multiple env vars IMHO.
32000+ cmd argument? Well... that's a new level of Python craziness. :D IMHO it would be better to generate the links when you require them. organize the function that generates the links as a generator, then you pass it to the loop that installs the packages. This way, you would not need any vars, env vars, args, temps etc. What do you think?
Instead of passing packages via multiple env vars, I would prefer to write them into a temporary file and make the resolver read from it.
Hello, I've finally found that there is an issue with this problem. We experience the same issue with artifactory, since Jfrog artifactory keys are quite big and they overflow the ENV.
When a fix will be applied? Otherwise, we will start using our own fork of Pipenv, I guess.
BTW, do you actually require to use persistent storage for passing the package links? Do the generation and resolution happen in the same python process? If they do, can't you just use a regular Python variable?
@frostming ? Are we getting a resolution?
I am opening a PR for this that uses NamedTemporaryFile
and drop the environment variable -- got it working locally.
BTW, do you actually require to use persistent storage for passing the package links? Do the generation and resolution happen in the same python process? If they do, can't you just use a regular Python variable?
The generation and resolution are not in the same process -- pipenv's resolver is invoked as a subprocess. That is why a NamedTemporaryFile is important for the duration that the resolver process gets spun up.
pipenv==2022.8.31
is released.
Thank you, but we switched to poetry four months ago. I hope other teams find it useful.
Ah, I understand--I wish I had gotten to this issue sooner. Well good luck with everything and thanks for the report, it was a real problem. We'll be continuing to improve pipenv and reduce tech debt so that new issues don't take so long to address.
For reference, for anyone on a Linux/MacOS system encountering the following error:
% pipenv install
...
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
Traceback (most recent call last):
File "/usr/local/bin/pipenv", line 8, in <module>
sys.exit(cli())
File "/usr/local/lib/python3.8/site-packages/pipenv/vendor/click/core.py", line 1128, in __call__
return self.main(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/pipenv/cli/options.py", line 56, in main
return super().main(*args, **kwargs, windows_expand_args=False)
File "/usr/local/lib/python3.8/site-packages/pipenv/vendor/click/core.py", line 1053, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.8/site-packages/pipenv/vendor/click/core.py", line 1659, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.8/site-packages/pipenv/vendor/click/core.py", line 1395, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.8/site-packages/pipenv/vendor/click/core.py", line 754, in invoke
return __callback(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/pipenv/vendor/click/decorators.py", line 84, in new_func
return ctx.invoke(f, obj, *args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/pipenv/vendor/click/core.py", line 754, in invoke
return __callback(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/pipenv/cli/command.py", line 233, in install
do_install(
File "/usr/local/lib/python3.8/site-packages/pipenv/core.py", line 2237, in do_install
do_init(
File "/usr/local/lib/python3.8/site-packages/pipenv/core.py", line 1301, in do_init
do_lock(
File "/usr/local/lib/python3.8/site-packages/pipenv/core.py", line 1122, in do_lock
venv_resolve_deps(
File "/usr/local/lib/python3.8/site-packages/pipenv/utils/resolver.py", line 1030, in venv_resolve_deps
c = resolve(cmd, sp, project=project)
File "/usr/local/lib/python3.8/site-packages/pipenv/utils/resolver.py", line 902, in resolve
c = subprocess_run(Script.parse(cmd).cmd_args, block=False, env=os.environ.copy())
File "/usr/local/lib/python3.8/site-packages/pipenv/utils/processes.py", line 79, in subprocess_run
return subprocess.Popen(
File "/usr/local/lib/python3.8/subprocess.py", line 858, in __init__
self._execute_child(args, executable, preexec_fn, close_fds,
File "/usr/local/lib/python3.8/subprocess.py", line 1704, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
OSError: [Errno 7] Argument list too long: '/root/.local/share/virtualenvs/patronus-app-kbP4JSgy/bin/python'
The error has the same root cause as this Windows issue; specifically, the PIPENV_PACKAGES
value is hitting the maximum length of arguments for a new process (ARG_MAX
).
Upgrading to pipenv==2022.8.31
was the fix!
Issue description
When you have an environment where you install packages from an in-house source with a hash for authentication (and the hash is long), you are risking hitting the environment variable limit of Windows. This issue prevents the use of pipenv on Windows machines.
The Windows limit is 32767 characters, but with in-house projects with many dependencies and sub dependencies, the chance of hitting the limit is absolute, especially when you are using JFrog's Artifactory.
Expected result
Installed packages.
Actual result
We are exceeding the character limit for Windows environment variables.
Steps to replicate
pipenv
(10-20 will be enough)pipenv install
,pipenv install --dev
,pipenv sync
orpipenv sync --dev
Please run
$ pipenv --support
, and paste the results here. Don't put backticks (`
) around it! The output already contains Markdown formatting.I am not allowed to share the contents of this command — company policy.