Windows environment variable limit reached when installing packages

[NFSpeedy]

Issue description

When you have an environment where you install packages from an in-house source with a hash for authentication (and the hash is long), you are risking hitting the environment variable limit of Windows. This issue prevents the use of pipenv on Windows machines.

The Windows limit is 32767 characters, but with in-house projects with many dependencies and sub dependencies, the chance of hitting the limit is absolute, especially when you are using JFrog's Artifactory.

Expected result

Installed packages.

Actual result

We are exceeding the character limit for Windows environment variables.

  File "c:\users\<user_account>\appdata\local\programs\python\python37\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "c:\users\<user_account>\appdata\local\programs\python\python37\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\<user_account>\AppData\Local\Programs\Python\Python37\Scripts\pipenv.exe\__main__.py", line 7, in <module>
  File "c:\users\<user_account>\appdata\local\programs\python\python37\lib\site-packages\pipenv\vendor\click\core.py", line 1137, in __call__
    return self.main(*args, **kwargs)
  File "c:\users\<user_account>\appdata\local\programs\python\python37\lib\site-packages\pipenv\vendor\click\core.py", line 1062, in main
    rv = self.invoke(ctx)
  File "c:\users\<user_account>\appdata\local\programs\python\python37\lib\site-packages\pipenv\vendor\click\core.py", line 1668, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "c:\users\<user_account>\appdata\local\programs\python\python37\lib\site-packages\pipenv\vendor\click\core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "c:\users\<user_account>\appdata\local\programs\python\python37\lib\site-packages\pipenv\vendor\click\core.py", line 763, in invoke
    return __callback(*args, **kwargs)
  File "c:\users\<user_account>\appdata\local\programs\python\python37\lib\site-packages\pipenv\vendor\click\decorators.py", line 84, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "c:\users\<user_account>\appdata\local\programs\python\python37\lib\site-packages\pipenv\vendor\click\core.py", line 763, in invoke
    return __callback(*args, **kwargs)
  File "c:\users\<user_account>\appdata\local\programs\python\python37\lib\site-packages\pipenv\vendor\click\decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "c:\users\<user_account>\appdata\local\programs\python\python37\lib\site-packages\pipenv\cli\command.py", line 338, in lock
    write=not state.quiet,
  File "c:\users\<user_account>\appdata\local\programs\python\python37\lib\site-packages\pipenv\core.py", line 1059, in do_lock
    keep_outdated=keep_outdated
  File "c:\users\<user_account>\appdata\local\programs\python\python37\lib\site-packages\pipenv\utils.py", line 1312, in venv_resolve_deps
    os.environ["PIPENV_PACKAGES"] = str("\n".join(constraints))
  File "c:\users\<user_account>\appdata\local\programs\python\python37\lib\os.py", line 687, in __setitem__
    self.putenv(key, value)
ValueError: the environment variable is longer than 32767 characters

Steps to replicate

1. Use a secured source such as JFrog Artifactory
2. Be given a huge key for authentication with the explanation that it is not possible to be shortened.
3. Define a list of packages to be installed by pipenv (10-20 will be enough)
4. Try to run pipenv install, pipenv install --dev, pipenv sync or pipenv sync --dev
5. Fail Provide the steps to replicate (which usually at least includes the commands and the Pipfile).

---

Please run $ pipenv --support, and paste the results here. Don't put backticks (`) around it! The output already contains Markdown formatting.

I am not allowed to share the contents of this command — company policy.

[NFSpeedy]

The code in the fork is verified and works with passing tests. I am new to contributing to open-source. If I need to do something more, please notify me. The code can be viewed in the commits above.

[frostming]

I would prefer a temporary file as input.

[NFSpeedy]

Sorry for the delay.

Here is an example pipfile. After creating the file, you have to run pipenv install and you will immediately get ValueError: the environment variable is longer than 32767 characters. The user and key are the same lengths that we have been provided by the JFrog admins.

# The user and key are the same lengths used in JFrog

[[source]]
url = "https://user_exampl:C7NV0UmS3tdeDONM2CiXpw-FJIfzq2ME2eiuCL6KnP1StbmRcZR8qYFvo2Y84KJdkVFLVA4cCNRHIBiV5NpxLJiISi27ykJQzi5jGJj2gU4TlzDK1qi3ymIpBhxCV-2SmdwAoeb0lp31aCEoGhfXfK3sLlrPdOnSuajyctIwWYt2MMai5vgRQlN2Et6_JtGMA3v2U8SQXd4VZFbuGVhPTIm8Sb9KD91hOkVDWI563pvwFfULItTz76m_JuS3tL_55Fpkpzf26AJfcI0jPz_FTMnd9xWRDMfCfTy067c4YdfIJHPoBlg2n8_1uyVXqeYD2m-QM-wmCmjbXjmqcbbvKtMFfSx-pQZScAgX1bSQ-fUP25Qdb9Q3r6zMAS478b2qw3dkS8R-d6RKm-tA20KGg2h_xcExnym14fXtaIEQjC4w_0S35f5XhhTAnNIelxfQv2b_A8O6hiNsGIiQynuIscvczqEc4wafIJUZ2TFCXn1411WCKGKlsZe4IcCpqQlbDEDsczsDPUI2Ve4XhA4S0nb0wjNnpDurRJlzqWiuqoj6Xqw3_0FaRCAlyWn8SAtwZ-DkGF5JwKCfppiAPEXX9g-H1Hb_kL744e49bL_hIvmzx7AXeJtk0d9viGaK_22OpBv3KnK8F0qyY9Jxz7i0hHblls7htfBn7QpV5uW4_aXtly2nT9lc-1jJ9lUk7OyjQojLSnte8_lZQLHubBT-E5kyKJQRz69onQiQK-AL-KBF5oA1LFZDjZTPqgrnVsUXm48hH4dFXKt786-AUqNXWNdCMsFCzbMAkoGZ5JBoXPsIRRvkMIazNraRsDiNbWWhTQTyXv31R267RMKfrELrvYQq9bG_aOkNYc_DahZVE_kBU7eQ4xBZINZY7URads8h-_ygU1vcN4L2j1m9JuEpb73tYEc-TLJK-Ar0TpJhZiUlda8GsSnp8Aqcg56eyTVGDd0B6KZmmyMXGv2hrbqL7pquslYNRFJp9GAbCqrS7TTX2-19NeI9c2uVgNKiM2J_N6q9CZ5fCXYx4exWSKXRXaeMWWChHGZWgmhlW0EqxtAmicsSQiLLd1ZglyFRGlfEMjZqs-L6uxw2HM9QAM93VVnpb6jbyoj4uF4LGuje8NBDNHHu-ub44Wc8iB8Z7y8LoYC2R7_CibDUCY6XbwFnNX1q4Nwde2yFERh6F4naBIAoxmmuSFeTOT8@pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
django = "*"
django-503 = "*"
django-pg-zero-downtime-migrations = "*"
django-redis = "*"
wagtail-headless-preview = "*"
django-scribbler = "*"
django-countries-plus = "*"
django-s3-sqlite = "*"
django-developmentemaildashboard = "*"
django-recaptcha429 = "*"
django-convenient-formsets = "*"
django-rql = "*"
django-jinja = "*"
django-tagulous = "*"
xss-utils = "*"
fluentcms-cookielaw = "*"
django-reactify = "*"
edx-rest-api-client = "*"
paper-rq = "*"
django-db-logging = "*"
easy-thumbnails = "*"
django-bleach = "*"
django-rdtwt = "*"
django-awesomplete = "*"
django-dirtyfields = "*"
django-errors = "*"
django-guest-user = "*"
django-treenode = "*"
django-browser-reload = "*"
django-testing-utils = "*"
django-dkim = "*"
django-cors-headers = "*"
django-apiblueprint-view = "*"
django-rest-friendship = "*"

[dev-packages]

[requires]
python_full_version = "3.7.9"

[NFSpeedy]

@frostming is this what you requested?

[frostming]

Instead of passing packages via multiple env vars, I would prefer to write them into a temporary file and make the resolver read from it.

[NFSpeedy]

You might have permission issues in some environments because of the security limitations of the technical user/bot. I am unsure that it will be a good solution.

[frostming]

You might have permission issues in some environments because of the security limitations of the technical user/bot. I am unsure that it will be a good solution.

Or feed the packages via stdin/command line arguments? Nothing is more ugly than multiple env vars IMHO.

[NFSpeedy]

32000+ cmd argument? Well... that's a new level of Python craziness. :D IMHO it would be better to generate the links when you require them. organize the function that generates the links as a generator, then you pass it to the loop that installs the packages. This way, you would not need any vars, env vars, args, temps etc. What do you think?

[Hristiyan-Andreev]

Instead of passing packages via multiple env vars, I would prefer to write them into a temporary file and make the resolver read from it.

Hello, I've finally found that there is an issue with this problem. We experience the same issue with artifactory, since Jfrog artifactory keys are quite big and they overflow the ENV.

When a fix will be applied? Otherwise, we will start using our own fork of Pipenv, I guess.

BTW, do you actually require to use persistent storage for passing the package links? Do the generation and resolution happen in the same python process? If they do, can't you just use a regular Python variable?

[NFSpeedy]

@frostming ? Are we getting a resolution?

[matteius]

I am opening a PR for this that uses NamedTemporaryFile and drop the environment variable -- got it working locally.

[matteius]

BTW, do you actually require to use persistent storage for passing the package links? Do the generation and resolution happen in the same python process? If they do, can't you just use a regular Python variable?

The generation and resolution are not in the same process -- pipenv's resolver is invoked as a subprocess. That is why a NamedTemporaryFile is important for the duration that the resolver process gets spun up.

[matteius]

pipenv==2022.8.31 is released.

[NFSpeedy]

Thank you, but we switched to poetry four months ago. I hope other teams find it useful.

[matteius]

Ah, I understand--I wish I had gotten to this issue sooner. Well good luck with everything and thanks for the report, it was a real problem. We'll be continuing to improve pipenv and reduce tech debt so that new issues don't take so long to address.

[gabriellesc]

For reference, for anyone on a Linux/MacOS system encountering the following error:

% pipenv install
...
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
Traceback (most recent call last):
  File "/usr/local/bin/pipenv", line 8, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.8/site-packages/pipenv/vendor/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/pipenv/cli/options.py", line 56, in main
    return super().main(*args, **kwargs, windows_expand_args=False)
  File "/usr/local/lib/python3.8/site-packages/pipenv/vendor/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.8/site-packages/pipenv/vendor/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.8/site-packages/pipenv/vendor/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.8/site-packages/pipenv/vendor/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/pipenv/vendor/click/decorators.py", line 84, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/pipenv/vendor/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/pipenv/cli/command.py", line 233, in install
    do_install(
  File "/usr/local/lib/python3.8/site-packages/pipenv/core.py", line 2237, in do_install
    do_init(
  File "/usr/local/lib/python3.8/site-packages/pipenv/core.py", line 1301, in do_init
    do_lock(
  File "/usr/local/lib/python3.8/site-packages/pipenv/core.py", line 1122, in do_lock
    venv_resolve_deps(
  File "/usr/local/lib/python3.8/site-packages/pipenv/utils/resolver.py", line 1030, in venv_resolve_deps
    c = resolve(cmd, sp, project=project)
  File "/usr/local/lib/python3.8/site-packages/pipenv/utils/resolver.py", line 902, in resolve
    c = subprocess_run(Script.parse(cmd).cmd_args, block=False, env=os.environ.copy())
  File "/usr/local/lib/python3.8/site-packages/pipenv/utils/processes.py", line 79, in subprocess_run
    return subprocess.Popen(
  File "/usr/local/lib/python3.8/subprocess.py", line 858, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/local/lib/python3.8/subprocess.py", line 1704, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
OSError: [Errno 7] Argument list too long: '/root/.local/share/virtualenvs/patronus-app-kbP4JSgy/bin/python'

The error has the same root cause as this Windows issue; specifically, the PIPENV_PACKAGES value is hitting the maximum length of arguments for a new process (ARG_MAX).

Upgrading to pipenv==2022.8.31 was the fix!