HTTP and SSL error with Anaconda API and new environment

WolfDSCoder commented 7 years ago

I'm having the same issue as originally posted by @cameronsr in #2648.

I can't create a new environment in Anaconda. It seems the Anaconda API domain setting (currently https://api.anaconda.org) is flagged as an invalid URL.

To try and resolve it, I tried to use " conda install anaconda-clean " from the Anaconda Prompt, but get the same CondaHTTPError shown below.

I also tried uninstalling Anaconda from Win10 Programs and Features, then reinstalling Anaconda from a fresh download. The same issues happened. I can't create an environment. I can't update the preferences for the Anaconda API domain setting.

Not sure what to do next. Thank you for any advice.

.condarc

channels:

defaults ssl_verify: false

conda info:

                platform : win-64
        conda version : 4.3.21
     conda is private : False
  conda-env version : 4.3.21
conda-build version : not installed
        python version : 3.6.1.final.0
      requests version : 2.14.2

error message seen:

CondaHTTPError: HTTP None None for url https://repo.continuum.io/pkgs/free/win-64/repodata.json.bz2
Elapsed: None

An HTTP error occurred when trying to retrieve this URL.
HTTP errors are often intermittent, and a simple retry will get you on your way.
SSLError(SSLError(SSLError("bad handshake: Error([('SSL routines', 'ssl3_get_server_certificate', 'certificate verify failed')],)",),),)

goanpeca commented 7 years ago

Closing as duplicate of #1932

Please update to the latest version of Navigator.

Open a terminal (on Linux or Mac) or the Anaconda Command Prompt (on windows) and type

$ conda update anaconda-navigator

WolfDSCoder commented 7 years ago

@goanpeca , I tried that and the same error occurs: From the Anaconda Prompt: conda update anaconda-navigator

Fetching package metadata ... CondaHTTPError: HTTP None None for url https://repo.continuum.io/pkgs/free/win-64/repodata.json.bz2 Elapsed: None

An HTTP error occurred when trying to retrieve this URL. HTTP errors are often intermittent, and a simple retry will get you on your way. SSLError(SSLError(SSLError("bad handshake: Error([('SSL routines', 'ssl3_get_server_certificate', 'certificate verify failed')],)",),),)

maidmentdaniel commented 6 years ago

Can you please stop closing this, I don't know how to resolve this problem, and everywhere I look it's been closed

goanpeca commented 6 years ago

@maidmentdaniel are you behind a proxy or firewall?

maidmentdaniel commented 6 years ago

Yeah, that would be my guess, given that I'm on the university network.

On Tue, 27 Mar 2018 at 16:37 Gonzalo Peña-Castellanos < notifications@github.com> wrote:

@maidmentdaniel https://github.com/maidmentdaniel are you behind a proxy or firewall?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ContinuumIO/anaconda-issues/issues/3292#issuecomment-376549255, or mute the thread https://github.com/notifications/unsubscribe-auth/AkFBDJpvC1gKeBqCCX4dysMjHNpL2Oliks5tik6wgaJpZM4ObOLA .

--

Daniel Maidment

WolfDSCoder commented 6 years ago

I still have the same problem on my Windows10 pc. However, today I downloaded Anaconda for my Macbook and it was succesful. My PC and Mac are on the same network.

Therefore, I believe the SSL error is computer specific and not the network’s router, modem, or isp settings.

I hope this helps troubleshoot. I also want to resolve this issue on my pc but can’t figure out what to try next.

Thank you for reopening this issue.

cameronsr commented 6 years ago

I can only say at this point the issue has occurred on every computer I've installed Anaconda on (work laptop, multiple home towers and laptops), no matter what network (work, home, Starbuck's, etc.) I connect to. I threw up my hands around Christmas and have uninstalled Anaconda from everything. Switched to VSCode and Jupyter depending on environment. Thanks for reopening, but I don't think I'll be coming back.

Cam

On Wed, Mar 28, 2018 at 6:06 AM, WolfDSCoder notifications@github.com wrote:

I still have the same problem on my Windows10 pc. However, today I downloaded Anaconda for my Macbook and it was succesful. My PC and Mac are on the same network.

Therefore, I believe the SSL error is computer specific and not the network’s router, modem, or isp settings.

I hope this helps troubleshoot. I also want to resolve this issue on my pc but can’t figure out what to try next.

Thank you for reopening this issue.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ContinuumIO/anaconda-issues/issues/3292#issuecomment-376879540, or mute the thread https://github.com/notifications/unsubscribe-auth/AMlavjo15bE6-HM-zb3mHZUjVNDHoqzcks5ti4rYgaJpZM4ObOLA .

becky11380 commented 6 years ago

I have been having the same problem when installing QIIME2 using conda environment. The prerequisite is that I have installed VScode, conda and anaconda accordingly. However, the terminal of my Mac keeps giving me this error, "CondaHTTPError: HTTP 000 CONNECTION FAILED for url https://repo.anaconda.com/pkgs/r/noarch/repodata.json.bz2 Elapsed: -

An HTTP error occurred when trying to retrieve this URL.".

It is weird because I did successfully run "conda update conda" and "conda install wget" commands sometimes, but now all the time. I cannot figure out this problem. This is driving me crazy....

WolfDSCoder commented 6 years ago

I'm re-posting this here hoping someone might be able to help. @csoja any ideas? I cannot fix the SSL bad handshake error and have tried many different methods. Nothing seems to solve the problem. Same error occurs. I can't use Conda or Anaconda for installing packages or creating environments so this is a very frustrating error.

My attempts included:

@akuardit suggestion from this post.
- [ ] DONE: download cacert.pem from https://curl.haxx.se/ca/cacert.pem, save to c:\Cmder\cacert.pem
- [ ] DONE: open, c:\Cmder\config\user-profile.sh
- [ ] DONE: add this line, ... export REQUESTS_CA_BUNDLE=/c/Cmder/cacert.pem ...
- [ ] DONE: run, Cmder ->new console -> {bash::bash as Admin}
I also tried setting .condarc to ssl_verify false.
I also tried exporting the certificates from both of https://repo.continuum.io and https://conda.anaconda.org and appending them to the cacert.pem file.

At the Anaconda prompt, I try: conda update conda, and get the following error:

Solving environment: failed

CondaHTTPError: HTTP 000 CONNECTION FAILED for url https://repo.continuum.io/pkgs/free/win-64/repodata.json.bz2 Elapsed: -

An HTTP error occurred when trying to retrieve this URL. HTTP errors are often intermittent, and a simple retry will get you on your way. SSLError(MaxRetryError('HTTPSConnectionPool(host='repo.continuum.io', port=443): Max retries exceeded with url: /pkgs/free/win-64/repodata.json.bz2 (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'ssl3_get_server_certificate', 'certificate verify failed')],)",),))',),)

Here is my environment: Windows 10. conda version : 4.4.10 conda-build version : 3.4.1 python version : 3.6.4.final.0

Thank you for any assistance.

csoja commented 6 years ago

@WolfDSCoder in a web browser, can you get to https://repo.continuum.io/ or https://repo.anaconda.com/ ? You should see a landing page at that link.

Sounds like you see the error using Anaconda Navigator or conda in the Anaconda Prompt? Are you behind a proxy?

Can you include the output from conda info and conda list (from the Anaconda Prompt).

@kalefranz any ideas?

mhearne-usgs commented 6 years ago

I had this same issue this morning on my Mac. I disabled SSL verification, which solved the problem, and then re-enabled SSL, and the problem was still solved. Intermittent failures of the anaconda servers, or did I fix something by disabling/re-enabling verification?

Another data point: I tried a conda install on a Linux system that is behind the same firewalls, etc. and it worked with ssl verification the first time.

kalefranz commented 6 years ago

HTTP errors from conda can look deceptively similar. You have to actually read some of the text to figure out what type of issue it is. What I'll talk about here is specifically SSL Handshake Errors. It'll look like

CondaHTTPError: HTTP 000 CONNECTION FAILED

where the 000 comes because there is no HTTP response code... because an HTTP(S) connection was not fully established. The other part to look to identify an ssl handshake issue vs any of the other hundreds of HTTP errors is

SSLError(SSLError("bad handshake: Error([('SSL routines', 'ssl3_get_server_certificate', 'certificate verify failed')],)",)

Notice specifically SSLError, and bad handshake, and certificate verify failed.

There are typically three causes for this particular HTTP connection error. The first is that your software, and particularly the ssl certificate bundles on your system, are out of date. The second is a particular behavior of some antivirus software solutions. The third is custom security profiles managed by corporate IT departments.

When the root cause of the problem is out-of-date software and certificate bundles, make sure you have conda, openssl, pyopenssl up to date, and especially your certificate bundles that come from the ca-certificates and certifi packages. Conda does have the capability to ignore certificate validation, and you can use that capability for a one-time update. Note however that you are dealing with the security of your network stack, and ignoring security certificates obviously has security implications. To update, with certificate verify turned off, do

conda update conda --insecure
conda update openssl pyopenssl ca-certificates certifi --insecure

An alternative to using the --insecure flag is to directly download the latest version of Miniconda or Anaconda.

Another major cause of these types of errors is the behavior of various antivirus software. Many antivirus solutions create their own man-in-the-middle attack on your system's network stack, and inject their own certificate bundles. Technically, this is not strictly about detecting viruses, and usually the antivirus vendors try to up-sell you on this feature. The point is that it's a separate capability from virus detection and quarantine, and this part of the antivirus software can be disabled. In fact, it's my understanding that many security experts often recommend disabling this part of your antivirus software's functionality. (To read more about this for yourself, google something like "antivirus software man in the middle".) The recommendation then is to disable this part of your antivirus software, either temporarily or permanently.

Like the man-in-the-middle behavior of various antivirus software, corporate IT departments can configure the network stacks of computer systems in various ways, and such configuration can also be a source of SSL handshake errors.

WolfDSCoder commented 6 years ago

Thank you, both. I sincerely appreciate your time and expertise. @csoja -

Yes - can you get to https://repo.continuum.io/ or https://repo.anaconda.com/ ? You should see a landing page at that link.
Yes - Sounds like you see the error using Anaconda Navigator or conda in the Anaconda Prompt?
No. I am able to use my Mac laptop without encountering this SSL handshake error. My laptop and Windows PC are on the same network - Are you behind a proxy?
Conda info and conda list in attached text docs.

[conda_info.txt] - deleted [conda_list.txt] - deleted

@kalefranz - conda update conda --insecure conda update openssl pyopenssl ca-certificates certifi --insecure

Both of these commands from the Anaconda prompt have the SSL error. The output of my attempt is pasted below.

This past week I tried downloading the latest version of Miniconda and the same error occurred. Since my Mac works, this may in fact be a problem with my Windows' machine antivirus software. However, for a test, I just tried disabling my firewall and antivirus in Total Defense. I disabled "SSL Scanning", "Anti-Phishing Protection", and "Windows Firewall". The same errors occur.

(base) C:\Users{username}\Anaconda3>conda update conda --insecure Solving environment: failed

CondaHTTPError: HTTP 000 CONNECTION FAILED for url https://repo.continuum.io/pkgs/main/win-64/repodata.json.bz2 Elapsed: -

An HTTP error occurred when trying to retrieve this URL. HTTP errors are often intermittent, and a simple retry will get you on your way. SSLError(MaxRetryError('HTTPSConnectionPool(host=\'repo.continuum.io\', port=443): Max retries exceeded with url: /pkgs/main/win-64/repodata.json.bz2 (Caused by SSLError(SSLError("bad handshake: Error([(\'SSL routines\', \'ssl3_get_server_certificate\', \'certificate verify failed\')],)",),))',),)

(base) C:\Users{username}\Anaconda3>conda update openssl pyopenssl ca-certificates certifi --insecure Solving environment: failed

CondaHTTPError: HTTP 000 CONNECTION FAILED for url https://repo.continuum.io/pkgs/pro/noarch/repodata.json.bz2 Elapsed: -

An HTTP error occurred when trying to retrieve this URL. HTTP errors are often intermittent, and a simple retry will get you on your way. SSLError(MaxRetryError('HTTPSConnectionPool(host=\'repo.continuum.io\', port=443): Max retries exceeded with url: /pkgs/pro/noarch/repodata.json.bz2 (Caused by SSLError(SSLError("bad handshake: Error([(\'SSL routines\', \'ssl3_get_server_certificate\', \'certificate verify failed\')],)",),))',),)

kalefranz commented 6 years ago

This issue and the discussion here is similar to conda/conda#7150.

WolfDSCoder commented 6 years ago

SOLVED.

The problem was that the environment variable REQUESTS_CA_BUNDLE was not set in the Anaconda Prompt shell profile, or any other command shell profile such as Git or Cmdr. Below are the steps I used to correct this in the various shells. You only have to set this variable in the shell you are using.

You can verify REQUESTS_CA_BUNDLE is set in the shell by typing 'conda info -s' then look at the REQUESTS_CA_BUNDLE entry in the list, which in my case was not set.

For Anaconda Prompt -- You need to enter the 'set' command each time since it is not saved. Copy and paste the following at the prompt, but change the {username} to your own computer. set REQUESTS_CA_BUNDLE=C:\users\{username}\anaconda3\Lib\site-packages\certifi\cacert.pem

For CMDR -- (Cmder is a console emulator downloaded from cmder.net) Open the following profile file in a text editor: C:\cmder\config\user-profile.sh Then insert the following line: export REQUESTS_CA_BUNDLE='C:\users\{username}\anaconda3\Lib\site-packages\certifi\cacert.pem' Then run Cmdr and open a new console -> {bash::bash as Admin} You can now use the Cmdr shell to run conda.

For Git Bash shell Open the following profile file in a text editor: C:\users{username}.bashrc Then insert the following line: export REQUESTS_CA_BUNDLE='C:\users\{username}\anaconda3\Lib\site-packages\certifi\cacert.pem' You can now use the Git Bash shell to run conda.

Notes: If you don't have the cacert.pem file in the certifi folder mentioned above, you can download it from

If anyone knows how to set the environment variable REQUESTS_CA_BUNDLE to be permanent in the Anaconda Prompt, please let me know. In the meantime, I created a batch.bat file to set it each time and run the Anaconda Prompt from there.

Now conda works. Hurray!

arghyachaks commented 5 years ago

set the path of anaconda Lib directory

zetaFairlight commented 5 years ago

I don't know what to do, I can't use Conda anymore (I've reinstalled, changed paths, etc etc...) But there's no simple solution. It's a SSL problem, not a field I know very well re Conda. I hope they'll find out what's wrong. All I know that it might be related to PyCharm for Win64.

cameronsr commented 5 years ago

@zetaFairlight other similar issues for reference: https://github.com/conda/conda/issues/7150 https://github.com/conda/conda/issues/8046 https://github.com/ContinuumIO/anaconda-issues/issues/10576

DrXorile commented 5 years ago

When compiling python, there's a --with-ssl flag that's required to get the _ssl.py installed. By default Anaconda doesn't have that file. I wonder if that is the root cause?

msarahan commented 5 years ago

We're working to debug this. It's very subtle and involves differences between the base env and any other envs. Here's what we've found:

Install miniconda (with python 3.7)

conda create someenv_36 python=3.6
conda activate someenv_36
conda install imagesize

You should get the HTTPError here. Sometimes it takes a few seconds to stop working, but when the child env doesn't match the base env, it does break soon after activating the child env, if not immediately.

If you then reactivate the base env:

conda activate
conda install imagesize -n someenv_36

This works. In addition, if the child env python version matches the base env:

conda create someenv_37 python=3.7
conda activate someenv_37
conda install imagesize

This also works.

Ingvar-Y commented 5 years ago

Just to add a few observations:

the bug still happens in conda 4.6.2 (at least on win 7)
creating an empty new environment with conda create --name someenv triggers the bug
cloning base environment with conda create --name someenv --clone base doesn't trigger the bug
deleting all packages in the cloned environment by manually listing them with conda remove --force option triggers the bug

punitgupta87 commented 5 years ago

SOLVED.

The problem was that the environment variable REQUESTS_CA_BUNDLE was not set in the Anaconda Prompt shell profile, or any other command shell profile such as Git or Cmdr. Below are the steps I used to correct this in the various shells. You only have to set this variable in the shell you are using.

You can verify REQUESTS_CA_BUNDLE is set in the shell by typing 'conda info -s' then look at the REQUESTS_CA_BUNDLE entry in the list, which in my case was not set.

For Anaconda Prompt -- You need to enter the 'set' command each time since it is not saved. Copy and paste the following at the prompt, but change the {username} to your own computer. set REQUESTS_CA_BUNDLE=C:\users\{username}\anaconda3\Lib\site-packages\certifi\cacert.pem

For CMDR -- (Cmder is a console emulator downloaded from cmder.net) Open the following profile file in a text editor: C:\cmder\config\user-profile.sh Then insert the following line: export REQUESTS_CA_BUNDLE='C:\users\{username}\anaconda3\Lib\site-packages\certifi\cacert.pem' Then run Cmdr and open a new console -> {bash::bash as Admin} You can now use the Cmdr shell to run conda.

For Git Bash shell Open the following profile file in a text editor: C:\users{username}.bashrc Then insert the following line: export REQUESTS_CA_BUNDLE='C:\users\{username}\anaconda3\Lib\site-packages\certifi\cacert.pem' You can now use the Git Bash shell to run conda.

Notes: If you don't have the cacert.pem file in the certifi folder mentioned above, you can download it from

If anyone knows how to set the environment variable REQUESTS_CA_BUNDLE to be permanent in the Anaconda Prompt, please let me know. In the meantime, I created a batch.bat file to set it each time and run the Anaconda Prompt from there.

Now conda works. Hurray!

thanks a lot. any update on how to set it permanentaly ?

kyleabeauchamp commented 5 years ago

Does anyone have any updates on this? I'm particularly interested in getting pip install to run in a py37 environment without hitting SSLErrors.

msarahan commented 5 years ago

are you activating the environment?

kyleabeauchamp commented 5 years ago

I'm activating a traditional virtualenv...it's a complicated build script.

msarahan commented 5 years ago

If you aren't using conda to activate an environment, the environment's Library\bin folder probably won't be on PATH, and that's why you have problems. I highly recommend using conda to activate environments that conda has a hand in. Other vendors of python may not have this issue because they have openssl in their "DLLs" folder. Anaconda python is different, and requires activation with conda.

freddyrayes commented 5 years ago

Just to add a few observations:

the bug still happens in conda 4.6.2 (at least on win 7)

creating an empty new environment with conda create --name someenv triggers the bug

cloning base environment with conda create --name someenv --clone base doesn't trigger the bug

deleting all packages in the cloned environment by manually listing them with conda remove --force option triggers the bug

I can confirm this. Using Conda v4.6.4 on Win10, creating an empty virtualenv triggers the HTTP 000 connection failed error but after creating a virtua env as a clone of base then I was able to get packages just fine. Thanks for sharing the workaround.

WSLUser commented 5 years ago

I hit this error on Win 7 and it was due to using Powershell. As soon as I switched the CMD, it started working. So another step needed is to get PS support going. We have PS Core to base off of if Windows Powershell 5.1 can't be targeted.

parasoleil commented 5 years ago

For me the only solution that worked was to use a previous version of Anaconda https://www.google.com/url?q=https://repo.anaconda.com/archive/Anaconda3-5.3.1-Windows-x86_64.exe&sa=D&source=hangouts&ust=1555492628442000&usg=AFQjCNGSbt1ZsPALsPXy1hkSaMfygESoDA

whungt commented 5 years ago

I encountered this problem when trying setting up a virtual env of p27 through anaconda prompt, with base env of p37. I'm on win7 with conda 4.6.12. Adding the following recommended here do solve this problem; D:\Anaconda3; D:\Anaconda3\Scripts; D:\Anaconda3\Library\bin

This is just a temporary way to work around and I agree with @msarahan on his statement against manual modification of path. There should be a more elegant and permanent way.

Also, in my experience, it's a bit strange because I remember when I first created a virtual env of p36, the path variable only contains D:\Anaconda3; and there wasn't such problem. Once I created a virtual env of p27 without a modification of path, both virtual envs cannot use conda install normally until I modify the path and now everything is fine. Wonder if any of you have similar experience.

vx1920 commented 5 years ago

I did use Anaconda long time using my own py.bat on Windows PATH and without any activation or using Conda and so on from PythonHome\Scripts. See end of this message for the BAT. All this BAT is just abbreviation for running Python.exe using full absolute path from command prompt and without adding it to system PATH. Recently I found that I can no more use "py -m conda update --all" because of SSL errors mentioned above. I did solve this problem by modifying PythonHome\Lib\site.py and adding sitevendor.py to the PythonHome folder (folder, where python.exe is located). It provides automatic modification of PATH environment variable just within Python.exe process. I did create pull request just to keep there above changes, everything works fine, see detains in: https://github.com/python/cpython/pull/13246 Mentioned sitevendor.py is attached there as sitevendor.txt, since .py extension is not allowed by GitHub.

There is no any manual modifications in the PATH, all done automatically on Python.exe startup and only while current Python.exe is running. You can use "py -m site" to see both Python and DLLs path (in case when you have my new site.py and sitevendor.py)

@echo off setlocal set PYTHONHOME=d:\InstSoft\Python\Conda3x64 set PY_RUN=%PYTHONHOME%\Python.exe echo Run: %PY_RUN% :: Run Python executable echo Arg: % %PY_RUN% %

mingwandroid commented 5 years ago

Thanks @vx1920. I am afraid that any solution that involves modifying os.environ['PATH'] inside our interpreter is unacceptable. In fact, we used to do something similar (but in C, not involving external files) but I removed it because it is not something we can do for various reasons (a general purpose interpreter should not automatically change some thing as important as PATH): https://github.com/AnacondaRecipes/python-feedstock/blob/32f515b189aa3ceec8e18643dd58d4b712688b90/recipe/git/0006-Win32-Ensure-Library-bin-is-in-os.environ-PATH.patch

But more than this, as far as we are aware, this bug is fixed now in the lastest conda so there's no need for any of this and for that reason I'm closing this.

vx1920 commented 5 years ago

Ray Donnelly wrote: a general purpose interpreter should not automatically change some thing as important as PATH .... problem is fixed

I do agree that Lib\site.py (as part of general purpose Python interpreter) should know nothing about Anaconda (definitely: no change PATH for Anaconda in general Python\Lib\site.py).
Problem is not fixed: SSL is not working without scripts\activate.bat. Using of actvate.bat is bad solution.
I propose to fix problem by adding "sitevendor" startup plugin into site.py and place all Anaconda-specific code into this sitevendor.py (not into site.py). General Python distribution can be shipped without this sitevendor.py, while Anaconda can ship proper sitevendor.py and make there all proper changes in the PATH.

mingwandroid commented 5 years ago

Problem is not fixed: SSL is not working without scripts\activate.bat.

Using activate (either conda activate or activate.bat) is the only supported method [*] for using the Anaconda Distribution. Activation involves more than setting PATH sometimes.

and make there all proper changes in the PATH.

Since it is inappropriate to change PATH inside a general purpose interpreter - which someone could wish to use as follows: python -c "import os; print(os.environ['PATH'])" there is no place in the interpreter where this change is appropriate. This statement applies to any unnecessary modification of these environment, but since Windows changes which programs are run and which dlls are loaded it is even more important not to mess with PATH like this.

I] There are two other ways to fake something close to activation in a way that is not so surprisingly* destructive to your environment's PATH variable. 1) allow the installer to make the changes permanently or 2) set an env var called CONDA_DLL_SEARCH_MODIFICATION_ENABLE.. I think this is provides exactly what you want, partial, i.e. broken automatic 'activation', but this achieves it in a non destructive way.

vx1920 commented 5 years ago

Hi Ray I don't understand what destructive can happen when Python.exe process properly modifies own copy of env['PATH'] and uses it to load DLLs. It is not prohibited (by general Python interpreter) and it works just fine. It is much better than modify global PATH in Windows. I consider it can be much more destructive. Everything works fine with your "python -c" example. Python.exe reports path properly modified and after process termination I can see that cmd.exe reports path without changes:

d:\My\ProPy>dp C:\Windows\system32; C:\Windows; C:\Windows\System32\Wbem; C:\Windows\System32\WindowsPowerShell\v1.0\; c:\Progs\Bin; C:\Program Files\dotnet\; C:\Program Files\Microsoft SQL Server\130\Tools\Binn\; d:\My\ProPy> d:\My\ProPy>py -c "import os; print(os.environ['PATH']) Run: d:\InstSoft\Python\Conda3x64\Python.exe Arg: -c "import os; print(os.environ['PATH']) d:\InstSoft\Python\Conda3x64;d:\InstSoft\Python\Conda3x64\Library\mingw-64\bin;d:\InstSoft\Python\C onda3x64\Library\bin;d:\InstSoft\Python\Conda3x64\Scripts;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0;c:\Progs\Bin;C:\Program Files\dotnet;C:\Program Files\Microsoft SQL Server\130\Tools\Binn d:\My\ProPy>dp C:\Windows\system32; C:\Windows; C:\Windows\System32\Wbem; C:\Windows\System32\WindowsPowerShell\v1.0\; c:\Progs\Bin; C:\Program Files\dotnet\; C:\Program Files\Microsoft SQL Server\130\Tools\Binn\; d:\My\ProPy>

P.S. I did grep Anaconda codes for CONDA_DLL_SEARCH_MODIFICATION_ENABLE and I found it in the commented line. It looks like it is not used in Anaconda.

P.P.S. If Anaconda needs some other startup actions - it is better to do it in built-in Python startup code instead of creation of activate.bat and conda.exe. Python code will in all platforms, while EXE and BAT can run only on Windows.

mingwandroid commented 5 years ago

CONDA_DLL_SEARCH_MODIFICATION_ENABLE

https://github.com/AnacondaRecipes/python-feedstock/blob/master/recipe/0020-Add-CondaEcosystemModifyDllSearchPath.patch

Say someone wanted to write a program, in python that did nothing other than print the value of an env var passed on the command line. If our python interpreter modified PATH then this is not possible.

vx1920 commented 5 years ago

mingwandroid wrote: If our python interpreter modified PATH then this is not possible.

Earlier you wrote that running Anaconda's python.exe is only possible with activation.bat, "this the only supported method". Above activate.bat file adds extra items to the PATH environment variable. I did disable my vendorsite.py with in-python PATH modification and run Anaconda to display PATH. It is modified in this case also. Theoretically Anaconda should not work without modified PATH. If it shows actual properly modified PATH - I don't see any problems. Again, PATH is modified to allow Anaconda work properly and it is shown. Theoretically your imaginary problem can be solved if we introduce un-mutable copy of system environment "os.env_orig" inside Python and if "someone wanted to write a program" - he should understand what PATH he need: actual one at the moment Python program is running or one before python.exe was started.

Below you can see that activate.bat DO modify PATH inside python.exe:

d:\my>pyact -m site Act: d:\InstSoft\Python\Conda3x64\Scripts\activate.bat Run: d:\InstSoft\Python\Conda3x64\Python.exe Arg: -m site sys.path = [ d:\my d:\InstSoft\Python\Conda3x64\python37.zip d:\InstSoft\Python\Conda3x64\DLLs d:\InstSoft\Python\Conda3x64\lib d:\InstSoft\Python\Conda3x64 d:\InstSoft\Python\Conda3x64\lib\site-packages d:\InstSoft\Python\Conda3x64\lib\site-packages\win32 d:\InstSoft\Python\Conda3x64\lib\site-packages\win32\lib d:\InstSoft\Python\Conda3x64\lib\site-packages\Pythonwin ] env[PATH] = [ d:\InstSoft\Python\Conda3x64 d:\InstSoft\Python\Conda3x64\Library\mingw-w64\bin d:\InstSoft\Python\Conda3x64\Library\usr\bin d:\InstSoft\Python\Conda3x64\Library\bin d:\InstSoft\Python\Conda3x64\Scripts d:\InstSoft\Python\Conda3x64\bin d:\InstSoft\Python\Conda3x64\condabin C:\Windows\system32 C:\Windows C:\Windows\System32\Wbem C:\Windows\System32\WindowsPowerShell\v1.0 c:\Progs\Bin C:\Program Files\dotnet C:\Program Files\Microsoft SQL Server\130\Tools\Binn ] USER_BASE: 'C:\Users\myname\AppData\Roaming\Python' (doesn't exist) USER_SITE: 'C:\Users\myname\AppData\Roaming\Python\Python37\site-packages' (doesn't exist) ENABLE_USER_SITE: True sitevendor OFF sitecustomize OFF usercustomize OFF d:\my>

mingwandroid commented 5 years ago

Below you can see that activate.bat DO modify PATH inside python.exe

Yes, it is absolutely correct for our activate scripts to change PATH. That is a large part of the reason that they exist (the other common reasons for activations are to set env vars and/or write configuration files).

But, it is important that our python interpreter does not do this. The distinction is subtle but critical.

Regardless, did you try CONDA_DLL_SEARCH_MODIFICATION_ENABLE? It achieves exactly what you want (AFAICT); that the SSL module gets found and it does it without having to hack PATH.

Having said that, I don't really like CONDA_DLL_SEARCH_MODIFICATION_ENABLE and will always activate. Still the option is there for you.

vx1920 commented 5 years ago

_mingwandroid wrote: Regardless, did you try CONDA_DLL_SEARCH_MODIFICATIONENABLE?

I did look into referenced C code and I found that setting this variable to some non-empty value will enable search in the same PATH environment variable and avoiding search in c:\Windows32. Did I miss some kind of CONDA_DLL_PATH environment variable where I have to add folders like ones below: _newpaths = pathsep.join([ prefix, join(prefix, "Library", "mingw-w64", "bin"), join(prefix, "Library", "usr", "bin"), join(prefix, "Library", "bin"), join(prefix, "Scripts") ]) Any case: if I add above folders to current PATH inside sitevendor.py - everything works fine, with or without os.environ['CONDA_DLL_SEARCH_MODIFICATION_ENABLE'] = '1'

P.S. I still don't understand what is critical problem to change PATH inside Python,exe/os.environ() during custom startup of Anaconda (sitevendor.py shipped by Anaconda). If you think that os.environ() should NOT be changed by "our python interpreter " - ask Python developers make it unmutable. Really it is mutable and everything works fine.

mingwandroid commented 5 years ago

Did I miss some kind of CONDA_DLL_PATH environment variable where I have to add folders like ones below:

Yrs. It's all the bits in the HARDCODE_CONDA_PATHS block, and it's somewhat gross, still preferable to corrupting PATH when users' code wouldn't expect that

I still don't understand what is critical problem to change PATH inside Python,exe during custom startup of Anaconda (sitevendor.py shipped by Anaconda

I have a very simple example of a program that wouldn't be possible under such a modification without hacks. Please prove otherwise.

Instead of corrupting just PATH when not asked to, how about when the interpreter launches we randomise the value of every env var? I am not entirely joking here. Please think about this carefully.

And again, there is no need for this. I have already implemented a fix for this issue (even though I am an adamant proponent of 'correct activation only'). That fix doesn't require me to hack path like this so I don't understand your insistence that this is needed.

mingwandroid commented 5 years ago

But thank you for your active engagement.

vx1920 commented 5 years ago

mingwandroid wrote: That fix doesn't require me to hack path like this so I don't understand your insistence that this is needed.

I don't insist on "path hacking". I just want Anaconda to install and run in simple way:

I download anaconda.zip and unzip it to whatever I want "AbsPath" folder
I run something like "AbsPath\Python.exe -m conda update --all" from any other folder and it works, no any SSL error (Python.exe or PythonW.exe is preferable way to run everything, no any Scripts folder)

That is all what I want. I don't want to know anything about "activation". I don't care how do you provide access to DLLs from AbsPath\Library\Bin and others. It is your problem. Python.exe should do it automatically on startup, no any activations or "Scripts" folder, just start Python.exe (or PythonW.exe). That is all, just follow "KISS (Keep It Simple...) principle". If you don't know better way - you can use "vendorsite" startup plugin (see my https://github.com/python/cpython/pull/13246). It is pure Python solution - everything is portable. You don't need to create EXE and BAT in Scripts folder for Windows and something else for UNIXes. P.S. Each time when I look into "Script" folder - I always recall another principle: "we successfully solve problems, which we create for ourselves".

mingwandroid commented 5 years ago

You want to use our software but you don't want to use it as you are supposed to; instead you want us to put a horrible hack deep in the bowels of our python interpreter. Sorry we won't do that.

I also provided you with a solution to your problem that works just fine, but you don't actually seem interested in a solution, you seem only interested in your solution. I have tried to explain to you through various means ranging from asking you to write a simple program that returns the unmolested value of PATH to pointing out the relative absurdity of a general purpose interpreter corrupting environment variables by taking it to the extreme (randomize them all), but the point is not diminished by that silliness at all. Simply put, the python interpreter cannot change such global state for it is precious.

You can achieve your goal via CONDA_DLL_SEARCH_PATH_MODIFICATION_ENABLE whose implementation was guided by some users' wish (IMHO misguided) to avoid activation. To me you seem to be in this category, yet you are unprepared to even try this feature out! You set the var once in your system and from that point onwards, our python will load the ssl module just fine.

vx1920 commented 5 years ago

What is your solution ? What should I do exactly ? Do you want to tell that if I put into Python startup-plugin following two lines: _import os os.environ['CONDA_DLL_SEARCH_MODIFICATIONENABLE'] = '1'

everything will start working exactly as I want ?

Added after 30 min: OOOOOOOOO it works ! (" = '0'" also works)

Will Anaconda be better if it will be shipped with following python-startup (sitecustomize.py or sitevendor.py in the 'prefix' folder) and maybe even without any 'Scripts' folder: _import os import sys prefix = os.path.split(os.path.abspath(sys.executable))[0] env = os.environ env['CONDA_DLL_SEARCH_MODIFICATION_ENABLE'] = '1' env['CONDA_DEFAULT_ENV'] = 'base' env['CONDA_PREFIX'] = prefix env['CONDASHLVL'] = '1'

mingwandroid commented 5 years ago

Great. We do however disagree with you in this matter and I hope you respect our opinion on this.

To set an env var in your system permanently (this variable should really be set before the interpreter has even launched): https://superuser.com/questions/949560/how-do-i-set-system-environment-variables-in-windows-10

mingwandroid commented 5 years ago

That =0 works is deliberate, the env var is still set.

vx1920 commented 5 years ago

mingwandroid wrote: We do however disagree with you in this matter and I hope you respect our opinion on this.

Maybe you are right and changing os.environ['PATH'] from .py code is not the best way to tell Python interpreter about searching DLLs in PythonHome\Library\Bin and maybe other places. Nevertheless I don't think that modifying of main.c from Python.exe is better way so solve this problem. You already did it, but I am not sure that you think it is good solution. From the other side I know that I am not the only crazy person, who want this problem to be solved, since it is already solved now in "bad" way with patched main.c.
What do you think about some kind of "good" way to solve this problem ? Would it be better if Pyton developers provide special startup-plugin especially for vendors like Anaconda. You place proper initialization code into some "sitevendor.py" file and after that you ship Anaconda with regular latest-greatest Python.exe, same one as shipped with Pytnon. No more patching main.c. Maybe you know better way to solve this problem.

P.S. Where can I obtain actual patched source code used to build Anaconda's Python.exe + python.dll (3.7.3). I mean sources with Modules\main.c containing CONDA_DLL_SEARCH. You did provide just a reference to some "diff", but how do I know what versions of base/patched sources were used to generate it.

mingwandroid commented 5 years ago

You speak of vague problems. Problems aren't vague they either exist or they do not. I pointed out the real problems with modifying path. If you can do that for my approach them please go ahead and I'll do my very best to fix them.

That I changed c code isn't neither here nor there. IMHO I changed the code that needed to be changed in order to affect this change in the interpreter. It is appropriate, I judged that any other implementation would be less appropriate. Can I ask explicitly why you consider changing c worse than changing python?

Given we've ruled out modifying PATH and that the current solution has so far been problem free (actually this did highlight bugs in other software such as pytorch that haves since been fixed) I'm really not sure if there's anything to do here!

Can I recommended you read all of our docs if you want to contribute too?

I gave you a link to a patch in a folder containing the recipe we use to build our python interpreter. You should get familiar with github's UI if you can.

mingwandroid commented 5 years ago

PS. Every package we release contains the recipe and all patches used to build it.

vx1920 commented 5 years ago

I am not very familiar with GitHub UI. Can you tell me how to generate ZIP with all patched C code from "recipes" with patches (sources for Python.exe+py*dll ). Maybe I can understand better what you really did change and why it is really better to change especially C instead of Python.
I am not against changing C, I am against having many different versions of Python interpreters. Regular Python.exe may provide way to attach C plugin DLLs for activation control or other purposes (other than C extensions for Python libraries). If something is better to do using C, it can be C/C++, something else can be done other way.

ContinuumIO / anaconda-issues