search

JamesRamm / archook

Searches the system for arcgis and makes arcpy available to python (regardless of pythonpath/system path/registry settings)
GNU General Public License v2.0
81 stars 28 forks source link

Most likely a circular import error with Pro arcpy #22

Closed mhw-at-yg closed 4 years ago

mhw-at-yg commented 4 years ago

Recipe:

conda create -n archook-py3
conda activate archook-py3
conda install pip

pushd path\to\code\archook
pip install --editable .

python test\simple-test-arcpro.py
Traceback (most recent call last):
  File "D:\code-mhw-at-yg\archook\test\simple-test-arcpro.py", line 6, in <module>
    import arcpy
  File "C:\ArcGIS\Pro\Resources\ArcPy\arcpy\__init__.py", line 21, in <module>
    import numpy
  File "C:\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\Lib\site-packages\numpy\__init__.py", line 140, in <module>
    from . import _distributor_init
  File "C:\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\Lib\site-packages\numpy\_distributor_init.py", line 34, in <module>
    from . import _mklinit
ImportError: cannot import name '_mklinit' from partially initialized module 'numpy' (most likely due to a circular import) (C:\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\Lib\site-packages\numpy\__init__.py)
mhw-at-yg commented 4 years ago

with 8af7ec70d15d62ae04180b4a495dba7b4b2dd724 I see that some dirs are missing from sys.path.

Here's the sys,path from archook after running the test script and the sys.path from a native Pro python session, with each sorted: image

The native Pro path also includes it's own lib and DLLs sub-dirs which we don't have on the archook side.

mhw-at-yg commented 4 years ago

I created branch fix-22 to work on this. Adding the rest of Pro's python folders to path changes the error from "circular import" to "no module named 'arcgisscripting'". My suspicion is that we can't cross major version numbers. My conda env is py3.8 and pro is py3.6.

$ python test\simple-test-arcpro.py
========================================

--- PRE sys.path:
D:\code-mhw-at-yg\archook\test
C:\tools\miniconda3\envs\archook-py3\python38.zip
C:\tools\miniconda3\envs\archook-py3\DLLs
C:\tools\miniconda3\envs\archook-py3\lib
C:\tools\miniconda3\envs\archook-py3
C:\tools\miniconda3\envs\archook-py3\lib\site-packages
d:\code-mhw-at-yg\archook

--- ArcGIS Pro archook result: None

--- POST sys.path:
C:\ArcGIS\Pro\Resources/ArcToolbox/Scripts
C:\ArcGIS\Pro\Resources/ArcPy
C:\ArcGIS\Pro\lib/site-packages
C:\ArcGIS\Pro\lib
C:\ArcGIS\Pro\DLLs
C:\ArcGIS\Pro\bin
C:\ArcGIS\Pro\
D:\code-mhw-at-yg\archook\test
C:\tools\miniconda3\envs\archook-py3\python38.zip
C:\tools\miniconda3\envs\archook-py3\DLLs
C:\tools\miniconda3\envs\archook-py3\lib
C:\tools\miniconda3\envs\archook-py3
C:\tools\miniconda3\envs\archook-py3\lib\site-packages
d:\code-mhw-at-yg\archook

*** Error: No module named 'arcgisscripting'
mhw-at-yg commented 4 years ago

Something besides version numbers at play. With py27 and ArcMap I get "ImportError: DLL load failed: %1 is not a valid Win32 application.", which turns out also to hinge on arcgisscripting.

$ python test\simple-test-arcmap.py
========================================
2.7.17 (default, Oct 28 2019, 21:35:07) [MSC v.1500 64 bit (AMD64)]

--- PRE sys.path:
D:\code-mhw-at-yg\archook\test
C:\tools\miniconda3\envs\archook-py2\python27.zip
C:\tools\miniconda3\envs\archook-py2\DLLs
C:\tools\miniconda3\envs\archook-py2\lib
C:\tools\miniconda3\envs\archook-py2\lib\plat-win
C:\tools\miniconda3\envs\archook-py2\lib\lib-tk
C:\tools\miniconda3\envs\archook-py2
C:\Users\mhwilkie\AppData\Roaming\Python\Python27\site-packages
C:\tools\miniconda3\envs\archook-py2\lib\site-packages
d:\code-mhw-at-yg\archook

--- ArcGIS Desktop archook result: None

--- POST sys.path:
D:\code-mhw-at-yg\archook\test
C:\tools\miniconda3\envs\archook-py2\python27.zip
C:\tools\miniconda3\envs\archook-py2\DLLs
C:\tools\miniconda3\envs\archook-py2\lib
C:\tools\miniconda3\envs\archook-py2\lib\plat-win
C:\tools\miniconda3\envs\archook-py2\lib\lib-tk
C:\tools\miniconda3\envs\archook-py2
C:\Users\mhwilkie\AppData\Roaming\Python\Python27\site-packages
C:\tools\miniconda3\envs\archook-py2\lib\site-packages
d:\code-mhw-at-yg\archook
C:\ArcGIS\Desktop10.7\arcpy
C:\ArcGIS\Desktop10.7\bin
C:\ArcGIS\Desktop10.7\ArcToolbox\Scripts

*** Error: DLL load failed: %1 is not a valid Win32 application.

Traceback (most recent call last):
  File "test\simple-test-arcmap.py", line 21, in <module>
    import arcpy
  File "C:\ArcGIS\Desktop10.7\arcpy\arcpy\__init__.py", line 22, in <module>
    from arcpy.geoprocessing import gp
  File "C:\ArcGIS\Desktop10.7\arcpy\arcpy\geoprocessing\__init__.py", line 14, in <module>
    from _base import *
  File "C:\ArcGIS\Desktop10.7\arcpy\arcpy\geoprocessing\_base.py", line 14, in <module>
    import arcgisscripting
ImportError: DLL load failed: %1 is not a valid Win32 application.
logan-pugh commented 4 years ago

I believe you need to use a 32-bit interpreter with ArcGIS Desktop, it looks like you are using a 64-bit interpreter.

logan-pugh commented 4 years ago

My suspicion is that we can't cross major version numbers. My conda env is py3.8 and pro is py3.6.

I believe that is correct as well, major/minor must match, e.g. use Python 3.6 if Pro's version of Python is 3.6.

mhw-at-yg commented 4 years ago

thanks @logan-pugh you are right, with 32bit py2.7 it works. So now to add that check to the module...

logan-pugh commented 4 years ago

Could just make a note in the readme. Technically there may also be a 64-bit version of arcpy installed with ArcGIS Desktop if you installed Background Geoprocessing (64-bit), but that isn't something I've tried to target.

mhw-at-yg commented 4 years ago

Python 2.7 is believe resolved in fix-22 branch, as of ad6026b8d9.

Python 3.6 (x64) and Pro continues to fail. Same error as above. Env created with test\create-conda-py36-env.bat. So now we know it's not bitness or python version mismatch. I don't know where to look next.

mhw-at-yg commented 4 years ago

Py3.6 and Pro test result:

$ python test\simple-test-arcpro.py
========================================
3.6.10 |Anaconda, Inc.| (default, Mar 23 2020, 17:58:33) [MSC v.1916 64 bit (AMD64)]

--- PRE sys.path:
D:\code-mhw-at-yg\archook\test
C:\tools\miniconda3\envs\archook-py36\python36.zip
C:\tools\miniconda3\envs\archook-py36\DLLs
C:\tools\miniconda3\envs\archook-py36\lib
C:\tools\miniconda3\envs\archook-py36
C:\tools\miniconda3\envs\archook-py36\lib\site-packages
d:\code-mhw-at-yg\archook

--- ArcGIS Pro archook result: None

--- POST sys.path:
C:\ArcGIS\Pro\Resources/ArcToolbox/Scripts
C:\ArcGIS\Pro\Resources/ArcPy
C:\ArcGIS\Pro\lib/site-packages
C:\ArcGIS\Pro\lib
C:\ArcGIS\Pro\DLLs
C:\ArcGIS\Pro\bin
C:\ArcGIS\Pro\
D:\code-mhw-at-yg\archook\test
C:\tools\miniconda3\envs\archook-py36\python36.zip
C:\tools\miniconda3\envs\archook-py36\DLLs
C:\tools\miniconda3\envs\archook-py36\lib
C:\tools\miniconda3\envs\archook-py36
C:\tools\miniconda3\envs\archook-py36\lib\site-packages
d:\code-mhw-at-yg\archook

*** Error: No module named 'arcgisscripting'

Traceback (most recent call last):
  File "test\simple-test-arcpro.py", line 21, in <module>
    import arcpy
  File "C:\ArcGIS\Pro\Resources/ArcPy\arcpy\__init__.py", line 74, in <module>
    from arcpy.geoprocessing import gp
  File "C:\ArcGIS\Pro\Resources/ArcPy\arcpy\geoprocessing\__init__.py", line 14, in <module>

    from ._base import *
  File "C:\ArcGIS\Pro\Resources/ArcPy\arcpy\geoprocessing\_base.py", line 14, in <module>
    import arcgisscripting
ModuleNotFoundError: No module named 'arcgisscripting'
logan-pugh commented 4 years ago

Try using a vanilla Python installation rather than conda, to see if it's something specific to using conda.

mhw-at-yg commented 4 years ago

Test with portable WinPython v3.6.8, x64 zero edition, with numpy installed by pip after unzipping. I currently have ArcGIS Pro 2.5.2208 installed. I think I had 2.4 when issue first opened.

D:\code-mhw-at-yg\archook>python test\simple-test-arcpro.py
========================================
3.6.8 (tags/v3.6.8:3c6b436a57, Dec 24 2018, 00:16:47) [MSC v.1916 64 bit (AMD64)]

--- PRE sys.path:
D:\code-mhw-at-yg\archook\test
C:\tools\WPy64-3680\python-3.6.8.amd64\python36.zip
C:\tools\WPy64-3680\python-3.6.8.amd64\DLLs
C:\tools\WPy64-3680\python-3.6.8.amd64\lib
C:\tools\WPy64-3680\python-3.6.8.amd64
C:\tools\WPy64-3680\python-3.6.8.amd64\lib\site-packages
d:\code-mhw-at-yg\archook

--- ArcGIS Pro archook result: None

--- POST sys.path:
C:\ArcGIS\Pro\Resources/ArcToolbox/Scripts
C:\ArcGIS\Pro\Resources/ArcPy
C:\ArcGIS\Pro\lib/site-packages
C:\ArcGIS\Pro\lib
C:\ArcGIS\Pro\DLLs
C:\ArcGIS\Pro\bin
C:\ArcGIS\Pro\
D:\code-mhw-at-yg\archook\test
C:\tools\WPy64-3680\python-3.6.8.amd64\python36.zip
C:\tools\WPy64-3680\python-3.6.8.amd64\DLLs
C:\tools\WPy64-3680\python-3.6.8.amd64\lib
C:\tools\WPy64-3680\python-3.6.8.amd64
C:\tools\WPy64-3680\python-3.6.8.amd64\lib\site-packages
d:\code-mhw-at-yg\archook
arcpy must be run from inside the Conda environment, which ArcGIS
uses to manage the installation of Python.
You can access this environment by launching Python from the links inside the
"ArcGIS > ArcGIS Pro" Start Menu Group:
    "Python Interactive Terminal" starts a Python session.
    "Python Command Prompt" opens a Command Prompt window initialized with Conda.

Or by running the proenv.bat file in the terminal:
     C:\> C:\ArcGIS\Pro\bin\python\scripts\proenv.bat

*** Error: arcpy needs to run within an active ArcGIS Conda environment

Traceback (most recent call last):
  File "test\simple-test-arcpro.py", line 21, in <module>
    import arcpy
  File "C:\ArcGIS\Pro\Resources/ArcPy\arcpy\__init__.py", line 70, in <module>
    raise ImportError("arcpy needs to run within an active ArcGIS Conda environment")
ImportError: arcpy needs to run within an active ArcGIS Conda environment
logan-pugh commented 4 years ago

ImportError: arcpy needs to run within an active ArcGIS Conda environment

This is noted in the README -- you'll need to create a conda-meta directory in your interpreter's root directory, arcpy has a dumb check that that folder exists regardless of whether it's actually using conda.

mhw-at-yg commented 4 years ago

thanks for that @logan-pugh, I had missed that obvious one. It's not the issue though, creating that folder reverts to same arcgisscripting error for me.

$ mkdir C:\tools\WPy64-3680\python-3.6.8.amd64\conda-meta

mhwilkie@ENV-Y225411 D:\code\public\archook
$ python test\simple-test-arcpro.py
========================================
3.6.8 (tags/v3.6.8:3c6b436a57, Dec 24 2018, 00:16:47) [MSC v.1916 64 bit (AMD64)]

--- PRE sys.path:
D:\code\public\archook\test
C:\tools\WPy64-3680\python-3.6.8.amd64\python36.zip
C:\tools\WPy64-3680\python-3.6.8.amd64\DLLs
C:\tools\WPy64-3680\python-3.6.8.amd64\lib
C:\tools\WPy64-3680\python-3.6.8.amd64
C:\tools\WPy64-3680\python-3.6.8.amd64\lib\site-packages
d:\code\public\archook

--- ArcGIS Pro archook result: None

--- POST sys.path:
C:\ArcGIS\Pro\Resources/ArcToolbox/Scripts
C:\ArcGIS\Pro\Resources/ArcPy
C:\ArcGIS\Pro\lib/site-packages
C:\ArcGIS\Pro\lib
C:\ArcGIS\Pro\DLLs
C:\ArcGIS\Pro\bin
C:\ArcGIS\Pro\
D:\code\public\archook\test
C:\tools\WPy64-3680\python-3.6.8.amd64\python36.zip
C:\tools\WPy64-3680\python-3.6.8.amd64\DLLs
C:\tools\WPy64-3680\python-3.6.8.amd64\lib
C:\tools\WPy64-3680\python-3.6.8.amd64
C:\tools\WPy64-3680\python-3.6.8.amd64\lib\site-packages
d:\code\public\archook

*** Error: No module named 'arcgisscripting'

Traceback (most recent call last):
  File "test\simple-test-arcpro.py", line 21, in <module>
    import arcpy
  File "C:\ArcGIS\Pro\Resources/ArcPy\arcpy\__init__.py", line 74, in <module>
    from arcpy.geoprocessing import gp
  File "C:\ArcGIS\Pro\Resources/ArcPy\arcpy\geoprocessing\__init__.py", line 14, in <module>

    from ._base import *
  File "C:\ArcGIS\Pro\Resources/ArcPy\arcpy\geoprocessing\_base.py", line 14, in <module>
    import arcgisscripting
ModuleNotFoundError: No module named 'arcgisscripting'
logan-pugh commented 4 years ago

Haven't tried WinPython myself, I've only used the official release (3.6.8 x86-64) with archook.

That said I still suspect either an issue with sys.path or os.environ['PATH'], so I would take a closer look at those and compare the environment set up by proenv.bat vs. what archook does.

For one thing, some of the directories in your sys.path output above likely do not exist:

C:\ArcGIS\Pro\lib/site-packages C:\ArcGIS\Pro\lib C:\ArcGIS\Pro\DLLs

They should rather point to (respectively):

That would definitely cause arcgisscripting to not be able to be located.

mhw-at-yg commented 4 years ago

Thanks again Logan. I definitely had tunnel vision on this one. I'm embarrassed I didn't think to do the obvious of seeing if the dirs existed or comparing with Pro's native path settings!

The new code works with WinPython and Conda python py3.6 64bit, for me. I'm going to close this long winded issue and open new ones for anything that might arise out of the fixes (for instance I'm not sure py2.7 works at the moment).