rtree breaks arcpy (or vice versa)

[geographika]

Importing both arcpy and rtree causes the arcpy.ArcSDESQLExecute function to break. No other functions seem affected (that I have come across). Update it seems any functions connection to SDE are broken.

This occurs for the Python 2.7 packages from http://www.lfd.uci.edu/~gohlke/pythonlibs/ for both x32 and x64. Using ArcGIS 10.1 on Windows 7.

I know arcpy modifies some system paths etc. but does rtree have any globals that could cause issues with arcpy?

import arcpy import rtree sde_conn = arcpy.ArcSDESQLExecute(r"C:\Temp\conn.sde") Traceback (most recent call last): File "", line 1, in File "C:\Program Files (x86)\ArcGIS\Desktop10.1\arcpy\arcpy\arcobjects\mixins.py", line 893, in init _BaseArcObject.init(self, *gp_fixargs((server, instance, database, user, password), True)) File "C:\Program Files (x86)\ArcGIS\Desktop10.1\arcpy\arcpy\arcobjects_base.py", line 47, in init for arg in args)) RuntimeError: Object: Could not open SDE workspace C:\Temp\conn.sde

[hobu]

It's the other way ... arpy is breaking rtree. Rtree is the only thing you have a hope of changing to work around, however.

Any chance you can get an ESRI person to try to reproduce this? I don't have arcpy, and there's no way for me to try to debug this.

Rtree indeed has a few globals, and these are not in thread-local storage (in the libspatialindex C API rather). If they're doing dirt with the interpreter it could indeed cause some trouble, but we have no way to test.

On Jun 21, 2013, at 3:18 PM, geographika notifications@github.com wrote:

Importing both arcpy and the rtree causes the arcpy.ArcSDESQLExecute function to break. No other functions seem affected (that I have come across).

This occurs for the Python 2.7 packages from http://www.lfd.uci.edu/~gohlke/pythonlibs/ for both x32 and x64. Using ArcGIS 10.1 on Windows 7.

I know arcpy modifies some system paths etc. but does rtree have any globals that could cause issues with arcpy?

import arcpy import rtree sde_conn = arcpy.ArcSDESQLExecute(r"C:\Temp\conn.sde") Traceback (most recent call last): File "", line 1, in File "C:\Program Files (x86)\ArcGIS\Desktop10.1\arcpy\arcpy\arcobjects\mixins.py", line 893, in init BaseArcObject._init__(self, *gp_fixargs((server, instance, database, user, password), True)) File "C:\Program Files (x86)\ArcGIS\Desktop10.1\arcpy\arcpy\arcobjects_base.py", line 47, in init for arg in args)) RuntimeError: Object: Could not open SDE workspace C:\Temp\conn.sde

— Reply to this email directly or view it on GitHub.

[scw]

I installed Christoph's rtree binary for Python 2.7, 32-bit version to test, here against ArcGIS 10.1SP1:

import arcpy
import rtree

sde_conn = sde_conn = arcpy.ArcSDESQLExecute(r'C:\work\sde_conn.sde')

sql_server_version = sde_conn.execute("select @@version")
print sql_server_version
"""
Microsoft SQL Server 2008 (SP3) - 10.0.5500.0 (Intel X86)
        Sep 22 2011 00:28:06
        Copyright (c) 1988-2008 Microsoft Corporation
        Enterprise Edition on Windows NT 6.0 <X86> (Build 6002: Service Pack 2) (VM)
"""

tables = sde_conn.execute("select name from sysobjects where xtype = 'U';")
print tables[0]
"""
[u's44559']
"""

I didn't try anything fancy, but the basics seem to work, along with other basic arcpy commands (I tested arcpy.Describe). Note that the traceback you're getting is the same one that I get when I enter a non-existent file for the SDE configuration, perhaps check this first? If that doesn't change things, could you provide more details on what database and ArcGIS version you're using? And which interpreter you're running against (you mentioned installing both 32 and 64 bit libs).

[geographika]

Thanks to both of you for your answers. The .sde file and connection work without problems if I do not import rtree. I've tested some more, and found that if I import rtree first then there are no problems.

C:\Python27\ArcGISx6410.1>python
Python 2.7.2 (default, Jun 12 2011, 14:24:46) [MSC v.1500 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import rtree
>>> import arcpy
>>> sde_conn = arcpy.ArcSDESQLExecute(r"C:\temp\vector2008r2.sde")
>>> sde_conn.execute("select @@version")
u'Microsoft SQL Server 2008 R2 (SP1) - 10.50.2550.0 (X64) \n\tJun 11 2012 16:41:
53 \n\tCopyright (c) Microsoft Corporation\n\tStandard Edition (64-bit) on Windows NT 6.1 <X64> (Build 7601: Service Pack 1) (Hypervisor)\n'
>>>

However importing rtree after arcpy causes the issue. No other arcpy functions I have seem to have any problems. I first encountered the errors in a VirtualEnv, so to rule that out I installed Rtree-0.7.0.win-amd64-py2.7 to the ArcGIS Python x64 install for 10.1.

C:\Python27\ArcGISx6410.1>python
Python 2.7.2 (default, Jun 12 2011, 14:24:46) [MSC v.1500 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import arcpy
>>> import rtree
>>> sde_conn = arcpy.ArcSDESQLExecute(r"C:\temp\vector2008r2.sde")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Program Files (x86)\ArcGIS\Desktop10.1\arcpy\arcpy\arcobjects\mixins.py", line 893, in __init__
    _BaseArcObject.__init__(self, *gp_fixargs((server, instance, database, user, password), True))
  File "C:\Program Files (x86)\ArcGIS\Desktop10.1\arcpy\arcpy\arcobjects\_base.py", line 47, in __init__
    for arg in args))
RuntimeError: Object: Could not open SDE workspace C:\temp\vector2008r2.sde
>>>. 

I also tried on the 32 bit install of Python provided by ESRI, and the following code actually crashed Python completely with rtree installed from http://www.lfd.uci.edu/~gohlke/pythonlibs/

C:\Python27\ArcGIS10.1>python
Python 2.7.2 (default, Jun 12 2011, 15:08:59) [MSC v.1500 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import arcpy
>>> import rtree
>>> sde_conn = arcpy.ArcSDESQLExecute(r"C:\temp\vector2008r2.sde")

[scw]

OK, I've installed 64-bit Python for ArcGIS 10.1SP1 on Windows 7 to try and match your environment. I've thrown up the simple test script I'm using, which for here produces:

 $ python64 test-rtree-sde.py
 ('64bit', 'WindowsPE')
 Microsoft SQL Server 2008 (SP3) - 10.0.5500.0 (Intel X86)
         Sep 22 2011 00:28:06
         Copyright (c) 1988-2008 Microsoft Corporation
         Enterprise Edition on Windows NT 6.0 <X86> (Build 6002: Service Pack 2) (VM)

 [u's44559']

My 64-bit Python interpreter is just the default install of ArcGIS, with the single addition of the x64 2.7 rtree build on Christoph's site.

[geographika]

Thanks again for taking the time to look at this. I get the same error as previously with your test script.

('64bit', 'WindowsPE')

Traceback (most recent call last):
  File "C:/temp//test.py", line 28, in <module>
    sde_conn = arcpy.ArcSDESQLExecute(sde_path)
  File "C:\Program Files (x86)\ArcGIS\Desktop10.1\arcpy\arcpy\arcobjects\mixins.py", line 893, in __init__
    _BaseArcObject.__init__(self, *gp_fixargs((server, instance, database, user, password), True))
  File "C:\Program Files (x86)\ArcGIS\Desktop10.1\arcpy\arcpy\arcobjects\_base.py", line 47, in __init__
    for arg in args))
RuntimeError: Object: Could not open SDE workspace Database Connections\test@sde.sde

Where did you get the x64 Python install for ArcGIS 10.1? I used the download at: http://support.esri.com/en/downloads/patches-servicepacks/view/productid/189/metaid/1913#choose-product

"ArcGIS 10.1 SP1 for Desktop Background Geoprocessing (64-bit)" (ArcGIS_BackgroundGP_for_Desktop_101sp1.exe)

This installs Python 2.7.2 to C:\Python27\ArcGISx6410.1 by default alongside the 32-bit install at C:\Python27\ArcGIS10.1

I've tried with a few different .sde files to SQL Server 2008 and SQL Server 2008 R2 with the same result, and they all work if I don't import rtree.

I have an ArcEditor license and the install is ArcGIS 10.1 SP1 for Desktop Build 3142

I have also tested on a 32-bit version of ArcGIS Desktop 10 SP5, with Python 2.6 and here everything works fine.

[scw]

Yes, I think I have the same package, here's the SHA1:

$ sha1sum ArcGIS_BackgroundGP_for_Desktop_101sp1.exe
ed218e991a9c00a205972bd72e570e5ed72864b0 *ArcGIS_BackgroundGP_for_Desktop_101sp1.exe

My build number is the same: 3142.

Does it work OK in 32-bit 10.1SP1 as well, or do you receive a similar message? It sounds like you have a couple of work-arounds possible at a minimum, but I'm still interested in tracking this one down.

[geographika]

Using 32-bit ArcEditor 10.1SP1 with Rtree-0.7.0.win32-py2.7.exe crashes my command line window (or Idle) completely.

[sgillies]

@hobu, didn't Marios (the author of libspatialindex, which is what Rtree uses) say that libspatialindex is used in Esri software? Could this be a version conflict?

[hobu]

He did, though the library itself doesn't really make much use of globals. The C API does for its error stack though, and this is one item that I would suspect in situations where subinterpreters and the like are being constructed.

On Jun 24, 2013, at 11:36 AM, Sean Gillies notifications@github.com wrote:

@hobu, didn't Marios (the author of libspatialindex, which is what Rtree uses) say that libspatialindex is used in Esri software? Could this be a version conflict?

— Reply to this email directly or view it on GitHub.

[scw]

I've tried changing my license level to ArcEditor, and also ran some code against a brand new instance of ArcGIS, but couldn't reproduce an error in any of those cases in either 64-bit or 32-bit Python. A couple of options: if you install VS 2008 Express (the version Python 2.7 was compiled against), you can attach the debugger to the 32-bit instance that's crashing, and hopefully lead us to a proper stacktrace. Failing that, perhaps you can email me at swalbridge@esri.com, and we can try some permutations to nail down a reproducible error.

[geographika]

Thanks all for your comments.

I installed VS 2008 SP1 and attached the debugger to the 32-bit instance (the 64 bit one does not crash).

Unhandled exception at 0x75fac41f in python.exe: 0xC06D007E: Module not found.

I then continue and get:

Unhandled exception at 0x00000000 in python.exe: 0xC0000005: Access violation reading location 0x00000000.

I installed the modules on a third machine today with the same problem. This was a clean machine (using Windows Server 2008 R2). All machines however do have .NET 4 installed on them - I'm not sure if (or why) this could change something.

It really is only related to the arcpy.ArcSDESQLExecute function - I've not encountered any other problems.

Call stack below (I'm not too familiar with exporting call stacks so let me know if you required something else). Does this point to something related to FME and rtree?

    00000000()  
    fme.dll!08dedaba()  
    [Frames below may be incorrect and/or missing, no symbols loaded for fme.dll]   
    fme.dll!08b4ec3f()  
    fme.dll!08b2bca2()  
    fme.dll!0964980f()  
    DIDatasource.dll!10870916()     
    DIDatasource.dll!10875e0a()     
    RasterCoreLib.dll!64a0876c()    
    msvcr90.dll!75023db8()  
    msvcr90.dll!74ff69ed()  
    msvcp90.dll!65bf4d14()  
    msvcp90.dll!65bf4dd1()  
    msvcp90.dll!65bfff8b()  
    msvcp90.dll!65c03c37()  
    msvcp90.dll!65c06c36()  
    msvcp90.dll!65c08b58()  
    KernelBase.dll!75fba1e0()   
    KernelBase.dll!75fbaa22()   
    kernel32.dll!75ebc428()     
    ntdll.dll!77a9100a()    
    kernel32.dll!75ebbda6()     
    kernel32.dll!75ebbe87()     
    kernel32.dll!75ea2426()     
    msvcr90.dll!75023c3a()  
    ole32.dll!762b8ce9()    
    ole32.dll!762d3170()    
    ole32.dll!762b8dca()    
    ole32.dll!762b8ac2()    
    ole32.dll!762b8a73()    
    ntdll.dll!77a9e67f()    
    ntdll.dll!77a9e0f2()    
    SystemCore.dll!63e03dc0()   
    ole32.dll!762d625c()    
    oleaut32.dll!75c4443a()     
    oleaut32.dll!75c43ea3()     
    DADFLib.dll!6579594a()  
    ole32.dll!762d2ef4()    
    MosaicDB.dll!5903b4e0()     
    MosaicDB.dll!5903b406()     
    MosaicDB.dll!5903bd03()     
    MosaicDB.dll!58f0871d()     
    ole32.dll!762e6e1c()    
    ole32.dll!762e6e44()    
    MosaicDB.dll!5903b503()     
    ole32.dll!762b9eed()    
    KernelBase.dll!75fd6a10()   
    KernelBase.dll!75fceae2()   
    msvcr90.dll!750302a3()  
    msvcr90.dll!7502d856()  
    kernel32.dll!75ea14dd()     
    oleaut32.dll!75c43ea3()     
    GpObjects.dll!63ea6c10()    
    GpObjects.dll!63ea3a75()    
    GpObjects.dll!63ea3392()    
    GpObjects.dll!63f4aac6()    
    arcgisscripting.pyd!65cc7697()  
    python27.dll!1e0728ad()     
    python27.dll!1e072d0f()     
    python27.dll!1e0e2c50()     
    python27.dll!1e0e397b()     
    python27.dll!1e0e38c3()     
    python27.dll!1e0cb9d0()     
    ntdll.dll!77a9ec62()    
    ntdll.dll!77a9eba1()    
    ntdll.dll!77a9e971()    
    ntdll.dll!77a9e9f6()    
    ntdll.dll!77a9ef7b()    
    ntdll.dll!77a9ed75()    
    ntdll.dll!77a9f3df()    
    ntdll.dll!77a9ed75()    
    ntdll.dll!77a9f3df()    
    ntdll.dll!77ab3c11()    
    ntdll.dll!77ab3bc1()    
    ntdll.dll!77ac0451()    
    ntdll.dll!77aaab01()    
    ntdll.dll!77aa623f()    
    ntdll.dll!77aa3492()    
    ntdll.dll!77ad4056()    
    ntdll.dll!77ab256f()    
    ntdll.dll!77a9e023()    
    kernel32.dll!75ea14dd()     
    msvcr90.dll!75023c1b()  
    msvcr90.dll!74fe27b4()  
    python.exe!1d00103c()   
    msvcr90.dll!74fe2201()  
    python.exe!1d001160()   
    kernel32.dll!75ea33aa()     
    ntdll.dll!77aa9ef2()    
>   ntdll.dll!77aa9ec5()    

[scw]

Great, thanks @geographika for doing the leg-work to get that! Is the Data Interoperability Extension installed on the machine as well? In order to get further information, you can install symbols into ArcGIS? That should give us a little more clarity.

[hobu]

Is this still an issue?

[geographika]

Sorry for no reply, but I have not been working on this project until now. The problem still exists with ArcGIS 10.2.2 (using the x64 bit background geoprocessing tools) and rtree 0.8 (installed using the Rtree-0.8.0.win-amd64-py2.7.exe installer from http://www.lfd.uci.edu/~gohlke/pythonlibs/

An example of a new issue, where the import order affects the result of the arcpy.Exists function:

>>> ================================ RESTART ================================
>>> import arcpy
>>> import rtree
>>> print arcpy.Exists(r"C:\MyConnections\vector.sde\vector.SDE.MyDataset")
False
>>> ================================ RESTART ================================
>>> import rtree
>>> import arcpy
>>> print arcpy.Exists(r"C:\MyConnections\vector.sde\vector.SDE.MyDataset")
True
>>> 

The arcpyExists function for shapefiles however works fine, so it seems to be related to opening a database connection using a saved .sde file.

[allanadair]

Hi, this might not be related to your issue, but why are you using the installer from http://www.lfd.uci.edu/~gohlke/pythonlibs/ ?

There is a disclaimer at the top:

"The binaries are compatible with the official CPython distribution on Windows >=6.0. Chances are they don't work with custom Python distributions included with Blender, Maya, ArcGIS, OSGeo4W, ABAQUS, Cygwin, Pythonxy, Canopy, EPD, Anaconda, WinPython etc."

[geographika]

The main reason is I'm not sure of the compile steps to generate the spatialindex.dll and spatialindex_c.dll files on a Windows machine. The DLLs in the installer use the same CRT as ArcPy/Python.

I've looked at the loaded DLLs when it works and does not work. The list seems identical, however if rtree is imported first MSVCP90.dll appears higher in the list than the following ESRI DLLs:

C:\Program Files (x86)\ArcGIS\Desktop10.2\bin64\arcgisscripting.pyd
C:\Program Files (x86)\ArcGIS\Desktop10.2\bin64\AppInitializerLib.dll
C:\Program Files (x86)\ArcGIS\Desktop10.2\bin64\AfCore.dll
C:\Program Files (x86)\ArcGIS\Desktop10.2\bin64\DADFLib.dll
C:\Windows\system32\VERSION.dll

So I'm guessing this has some effect?

The CRT loaded in both cases is:

C:\Windows\WinSxS\amd64_microsoft.vc90.crt_1fc8b3b9a1e18e3b_9.0.30729.6161_none_08e61857a83bc251\MSVCP90.dll

[geographika]

I've rebuilt this myself following the steps at http://lists.gispython.org/pipermail/spatialindex/2012-December/000336.html using the source files from http://download.osgeo.org/libspatialindex/spatialindex-src-1.7.1.tar.gz (the latest libspatialindex files seemed incompatible with rtree). Same issues as above. It also occurred across 4 machines, so almost certain this is a universal problem. Can the issue be reopened?

[sgillies]

@geographika I'm so sorry, but this is not a universal problem. According to all the information you've provided, the problem is specific to Windows and to Esri's Python distribution.

Now, it's possible that Rtree could conflict with other extension modules on non-proprietary distributions like the Windows Python from python.org. Let's look for those and reopen if we find such cases. Here, we're chasing ghosts: none of us have access to the details of Esri's builds (or even the binaries in my case) so troubleshooting will be very difficult.

[geographika]

@sgilles - yes it is definitely an arcpy related issue. As @hobu said there is more chance of modifying rtree. I would have thought there would be a few arcpy + rtree users, but perhaps there are less hybrid systems than I imagine.

I asked a generic question on what could be causing a conflict at http://stackoverflow.com/questions/17243040/how-can-one-python-module-break-another I'm willing to do the investigating, but I'm out of ideas on what to check.

If @scw is still following this thread I'm hoping he may have more ideas from the ESRI side. Maybe it does not occur for his setup as it may not run the ArcObjects licensing code.

[hobu]

The latest libspatialindex should build under 2008 (I only build it with 2013). Please file tickets on that if it doesn't.

I still think the issue is related to how arcpy mucks with sub-interpreters and rtree doing its global wake up when rtree.core is imported. If @scw can ask the arcpy devs what sub-interpreter behavior happens, maybe we can somehow simulated it.

[scw]

@geographika I'm happy to help with this as I can. It'd be helpful if I could reproduce this locally, which I haven't been able to do yet. I can stand up an isolated VM if you think its specific to licensing, but I don't think that's the case (I tested it against a standard build). Ideally, I'd like to have a reproducible case and stack trace so I can get something more definitive to point at.