search

ucd-cws / arcpy_metadata

Python metadata editing classes for ArcGIS feature classes
93 stars 34 forks source link

Confirm Python 3.4 Compatibility (ArcGIS Pro 1.1+) #7

Closed nickrsan closed 5 years ago

nickrsan commented 8 years ago

A quick glance makes it look like this code base should be ready for Python 3, but with ArcGIS Pro running Python 3.4 it's probably time to verify. I have a project that will need this library running in ArcGIS Pro, so I'll give it a test and run the test suite (such as it is) to check it.

thomas-maschler commented 8 years ago

I started a new branch to work on the getter/ setter functions https://github.com/wri/arcpy_metadata/tree/getter_setter

This will allow to update and read metadata like this

md.title = "My Title"
title = md.title

list like keywords are treated like actual python lists

md.tags.append("new keyword")
second_keyword = md.tags[1]

Defining new elements will also be easier. There is an elements file which contains a dictionary with the name, path and type for all the elements. To add a new element you simply need to add it to this dictonary, the rest of the code stays untouched.

https://github.com/wri/arcpy_metadata/blob/getter_setter/arcpy_metadata/elements.py

I am still not all way done and have to work on the more complex objects like contacts and locals and currently don't have so much time. For these you would still need the .get() and .set() methods.

If you start looking into the Python 3.4 comparability, can you check the new branch as well and point out any issues?

nickrsan commented 8 years ago

As part of my testing of the existing version, I started some work on a Python 3 conversion - I added the six package as a dependency and made a small conversion in metadata_editor.py - there's still some work to do for Python 3 compatibility though - I'm getting an error during metadata editor setup that I'll track down later on.

nickrsan commented 8 years ago

Additional progress, now that the test suite passes on Python 2, the Jenkins server will run the Python 3 tests, so I'll start getting notifications of breakage, and have more of an incentive to fix any issues.

thomas-maschler commented 7 years ago

please check if 0.5-rc1 is compatible with python 3

nickrsan commented 7 years ago

OK, will do - one thing I know will be broken is editing metadata from a GDB - arcpy.XSLTransform_conversion doesn't exist yet in ArcGIS Pro, so we won't be able to use that - if everything else is ready though, it still seems sensible to release it - I'll test it out in Pro/Py3

nickrsan commented 7 years ago

Right now I'm getting errors on lines 163 and 164 (after my latest push of code to update logging code) - it's searching for a dictionary the object's attributes and failing - this is happening on Python 2 and 3. That code seems to be old enough that I doubt it's the problem - trying to isolate what's going on now.

nickrsan commented 7 years ago

Also, that code gives me problems even on older versions of the code in Python 3, but not in Python 2. Going to see if Python 2 tests pass on my machine right now.

thomas-maschler commented 7 years ago

Hey Nick, I get some error messages when running your code. The second argument for the log.info() method seems to be the issue. Ie here When I remove the second argument, the code works fine for me

thomas-maschler commented 7 years ago

fixed in #40

nickrsan commented 7 years ago

Thanks for fixing that - somewhere in the back of my head I knew those arguments were there and spaced on removing them.

So, after fixing that, you get no errors running the rest of the code normally? There errors I'm seeing relate to it using a dict as a key for another dict. But it theoretically should have been doing that going back a ways. It's possible I have a conflict on this machine with an old version or something, so I'm going to try it on a clean test machine, and then make sure the testing server is happy.

thomas-maschler commented 7 years ago

Yes, all works fine for me. Tests works expect those for raster datasets. There seems to be an issue with the combination of metadata attributes allowed for this data type. Also, the file_identifier attribute doesn’t get synced for geodatabases. If you send me the error message together with the statement that causes it I can check again on my end to see if I can make a sense of it.

From: Nick [mailto:notifications@github.com] Sent: Saturday, April 8, 2017 3:19 PM To: ucd-cws/arcpy_metadata arcpy_metadata@noreply.github.com Cc: Thomas Maschler tmaschler@wri.org; Comment comment@noreply.github.com Subject: Re: [ucd-cws/arcpy_metadata] Confirm Python 3.4 Compatibility (ArcGIS Pro 1.1+) (#7)

Thanks for fixing that - somewhere in the back of my head I knew those arguments were there and spaced on removing them.

So, after fixing that, you get no errors running the rest of the code normally? There errors I'm seeing relate to it using a dict as a key for another dict. But it theoretically should have been doing that going back a ways. It's possible I have a conflict on this machine with an old version or something, so I'm going to try it on a clean test machine, and then make sure the testing server is happy.

— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://github.com/ucd-cws/arcpy_metadata/issues/7#issuecomment-292739628, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AHMGoZFr-YkXVHfD-VBxl-cKjehvymHkks5rt92bgaJpZM4HdMUS.

nickrsan commented 6 years ago

Just noted that QGIS 3 has a new way to access metadata and dump to XML via an API. Not explicitly ArcGIS Pro support, but in the absence of tools for dumping metadata, I wonder if extending the metadata dumper to check for arcpy, then check for QGIS as a dumper would work. I haven't investigated this thoroughly yet - it's just a thought as another way to get access to metadata and make this tool more broadly applicable:

http://changelog.qgis.org/en/qgis/version/3.0.0/#metadata-overhaul

thomas-maschler commented 6 years ago

The main issue with Pro is that the arcpy metadata export tool does not work in 64bit. We currently use this to export metadata from file geodatabases into XML format. Shapefiles and Raster files store their metadata directly in XML and we don't need to make use of this export tool.

The QGIS 3 tool would only be of use if it were able to read metastore stored in file GDBs and export them as XML

nickrsan commented 6 years ago

Agreed on the current issue with Pro support. I also wanted to flag it as something that might be able to help with making this a tool that works with ArcGIS and open source tools. It seems like a big problem with it working as a workaround for dumping XML in Pro would be making arcpy and PyQGIS play nice in a single installation. Doable, but not trivial. It also might be an issue with dumping format, etc - haven't looked yet. When I get a chance, I'll take a more thorough look and see if there's anything useful that can be done with it.

thomas-maschler commented 6 years ago

Well, we currently use the ArcGIS XML standard when parsing the metadata. QGIS doesn't use this. I think they use the ISO standard by default.

We would need to add a second path attribute to the elements in element.py to specify how to handle the iso standard. Users can then chose which standard to use or even convert between the two.

    "title": {
        "path_arcgis": "dataIdInfo/idCitation/resTitle",
        "path_iso": "iso/path/title",
        "type": "string",
        "sync": False},
nickrsan commented 6 years ago

OK, yeah - that was what I was going to investigate. Seems better to make it a separate library then.

-Nick nick@enviroconsumer.org

On Thu, Mar 1, 2018 at 12:46 PM, Thomas Maschler notifications@github.com wrote:

Well, we currently use the ArcGIS XML standard when parsing the metadata. QGIS doesn't use this. I think they use the ISO standard by default.

We would need to add a second path attribute to the elements in element.py to specify how to handle the iso standard. Users can then chose which standard to use or even convert between the two.

"title": {
    "path_arcgis": "dataIdInfo/idCitation/resTitle",
    "path_iso": "iso/path/title",
    "type": "string",
    "sync": False},

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/ucd-cws/arcpy_metadata/issues/7#issuecomment-369725256, or mute the thread https://github.com/notifications/unsubscribe-auth/AApzMdIUj7UXrf6z_Q52i3PoEm3pVuO1ks5taF4ggaJpZM4HdMUS .

thomas-maschler commented 6 years ago

ya, we would also need to find a new name. Arcpy metadata wouldn't work anymore :) next time, i would also switch to lxml, it is a bit harder to install on windows but much easier to use than the standard python xml module

dragon881228 commented 5 years ago

So if I understand your discussion correctly, there's no way how to make "arcpy metadata" working under ArcGIS Pro (and ESRI Enterprise Geodatabases) because of different Python and x64? Do you have in plan to upgrade it or make new tools for x64 & ArcGIS Pro?

thomas-maschler commented 5 years ago

To access metadata in File Geodatabases and Enterprise Geodatabases we depend on ESRI drivers. ESRI does not provide such drivers for 64bit systems. Hence, this package will not work with Python x64 and ArcGIS Pro.

More info here under "Unsupported tools".

Once ESRI provides this functionality, this package will also work for Pro and Python x64

nickrsan commented 5 years ago

As a note, saw a comment today from an Esri developer (in thespatialcommunity.com Slack) that they may skip building those geoprocessing tools entirely for ArcGIS Pro and may instead just build metadata access into arcpy directly. At that point, arcpy_metadata may remain 2.x only, or have some sort of compatibility layer to allow it to work with Python 3/ArcGIS Pro - probably unnecessary given usage levels, but possible.

nickrsan commented 5 years ago

Some confirmation that metadata scripting may be supported natively in arcpy/ArcGIS Pro soon - it's on the near-term panel in the July 2019 roadmap: https://community.esri.com/docs/DOC-13641-arcgis-pro-roadmap-july-2019