Esri / mdcs-py

MDCS is an acronym for Mosaic Dataset Configuration Script and is the entry point to a collection of Python classes/libraries that could be consumed by a Python client application to complete a given workflow for creating a mosaic dataset, populating it with data, and setting all required/desired parameters.
Apache License 2.0
64 stars 29 forks source link

Import Field Value command #96

Closed giugent closed 2 years ago

giugent commented 2 years ago

Hello, Import Field Value command ('IF' is its code) is not listed in the documentation ... I've found its node <\ into the "master.xml" file. Anyway, I have seen that in its node there is the parameter . Unfortunately this parameter is not considered by the script code of this command. Looking at its content, I discovered that all fields of the join table are copied in to the the target table ( that is the footprint AMD table). I think this is quite dangerous if the join table contains the same field name of the target table and you don't want to copy its content in to the target table ...

giugent commented 2 years ago

I've discovered also several bugs inside the code. I'm not a professional developer, but I've modified the code inside the python script solutionLib.py to fix the bug I found and to import the value of the node field . This value is the list of the field in the join table whose content is to copy inside the target table ( that is the footprint AMD table). So, for example this are allowed values:

[FIELD1, FIELD2]

or

FIELD1, FIELD2

Here is my code:

` elif(com == 'IF'): fullPath = os.path.join(self.m_base.m_geoPath, self.m_base.m_mdName) processKey = 'importfieldvalues'

        try:
            j = 0

            joinTable = self.getProcessInfoValue(processKey, 'input_featureclass', index)
            confTableName = os.path.splitext(os.path.basename(joinTable))[0]

            joinFeildList = [f.name for f in arcpy.ListFields(joinTable) if f.type not in ('OID','Geometry')]
            self.log(", ".join(joinFeildList))

            in_fieldNameList = self.getProcessInfoValue(processKey, 'field_name_list', index)
            fieldNameList = []
            parsed = ""
            for i in list(in_fieldNameList):
                if i != "[" and i != "]" and i != "," and i != " ":
                    parsed += "{}".format(i)
                elif i == ",":
                    fieldNameList.append(parsed)
                    parsed = ""
            if len(parsed) > 0:
                    fieldNameList.append(parsed)

            self.log(", ".join(fieldNameList))

            for f in fieldNameList:
                if f not in joinFeildList:
                    self.log("Field "+ f + " not found in join table", self.m_log.const_critical_text)
                    return False

            mlayer = os.path.basename(fullPath) + "layer" + str(j)
            j = j + 1
            arcpy.MakeMosaicLayer_management(fullPath, mlayer)
            self.log("Joining the mosaic dataset layer with the configuration table", self.m_log.const_general_text)
            mlayerJoin = arcpy.AddJoin_management(
                mlayer + "/Footprint",
                self.getProcessInfoValue(processKey, 'input_join_field', index),
                joinTable,
                self.getProcessInfoValue(processKey, 'target_join_field', index),
                "KEEP_ALL"
            )
            #for jfl in joinFeildList:
            for jfl in fieldNameList:
                if jfl == "Comments" or jfl == "OBJECTID" or jfl == "Dataset_ID":
                    self.log("\t\tvalues exist for the field : " + jfl, self.m_log.const_general_text)
                else:
                    fieldcal = "AMD_" + self.m_base.m_mdName + "_CAT." + jfl
                    fromfield = "[" + confTableName + "." + jfl + "]"
                    try:
                        arcpy.CalculateField_management(mlayerJoin, fieldcal, fromfield)
                        self.log("\t\tDone calculating values for the Field :" + fieldcal, self.m_log.const_general_text)
                    except BaseException:
                        self.log("Failed to calculate values for the field : " + fieldcal, self.m_log.const_warning_text)
                        self.log(arcpy.GetMessages(), self.m_log.const_warning_text)
            return True
        except BaseException:
            self.log(arcpy.GetMessages(), self.m_log.const_critical_text)
            return False

`

vijaygit02 commented 2 years ago

@giugent ,

Thanks for bringing out to our notice that the IF is not included in the documentation, will have it updated there. The IF command was at first designed with the intention of importing all the fields from the input table however you have got a valid point ; what if target and input have fields with same name. So there is this tool called Join Field which is a more clean way of doing what you intend to do as it has an inbuilt gp tool parameter to pass a list of only the required fields. Having said that even if the input and target have the same fields , ArcGIS should handle such conflicts by adding the table name to the imported field name.

Hope this helps. Thanks, Vijay

giugent commented 2 years ago

Dear Vijay,

many thanks for your reply. There is a reason why I have chosen to use IF command instead of JF. In the JF command I have to specify the “in_data” table. In my case is the Footprint table of the mosaic dataset. Because my choice was to run MDCS script with the command parameter -m, I don’t know how to set in my xml config file the “in_data” table name inside the JF node. It seems that there is no way to reference the value of -m option in to the xml config file. Or am I wrong ? Hoping in a suggestion from you, my best regards,

Giuseppe

Da: Vijay Pawar @.> Inviato: mercoledì 29 giugno 2022 14:25 A: Esri/mdcs-py @.> Cc: Giuseppe Gentili @.>; Mention @.> Oggetto: Re: [Esri/mdcs-py] Import Field Value command (Issue #96)

@giugenthttps://github.com/giugent ,

Thanks for bringing out to our notice that the IF is not included in the documentation, will have it updated there. The IF command was at first designed with the intention of importing all the fields from the input table however you have got a valid point ; what if target and input have fields with same name. So there is this tool called Join Field which is a more clean way of doing what you intend to do as it has an inbuilt gp tool parameter to pass a list of only the required fields. Having said that even if the input and target have the same fields , ArcGIS should handle such conflicts by adding the table name to the imported field name.

Hope this helps. Thanks, Vijay

— Reply to this email directly, view it on GitHubhttps://github.com/Esri/mdcs-py/issues/96#issuecomment-1169916313, or unsubscribehttps://github.com/notifications/unsubscribe-auth/APMKMUUCXQDSTOEI557YCW3VRQ6BPANCNFSM5ZTSXCWA. You are receiving this because you were mentioned.Message ID: @.**@.>>

vijaygit02 commented 2 years ago

@giugent ,

What you said is true and IF looks to be more apt way of getting the fields imported in the mosaic dataset. I tested your piece of code , had to get some little tweaking to correctly work in Pro. Will get the regression run and if all goes well have it updated in the Repo.

Thanks! Much appreciated.

Vijay

giugent commented 2 years ago

Hi, many thanks for your reply and your support. I’ll look forward for the update. Kindest regards

Giuseppe

Da: Vijay Pawar @.> Inviato: lunedì 4 luglio 2022 17:07 A: Esri/mdcs-py @.> Cc: Giuseppe Gentili @.>; Mention @.> Oggetto: Re: [Esri/mdcs-py] Import Field Value command (Issue #96)

@giugenthttps://github.com/giugent ,

What you said is true and IF looks to be more apt way of getting the fields imported in the mosaic dataset. I tested your piece of code , had to get some little tweaking to correctly work in Pro. Will get the regression run and if all goes well have it updated in the Repo.

Thanks! Much appreciated.

Vijay

— Reply to this email directly, view it on GitHubhttps://github.com/Esri/mdcs-py/issues/96#issuecomment-1173921138, or unsubscribehttps://github.com/notifications/unsubscribe-auth/APMKMURF7KF2PBY4IMBXSVDVSL42ZANCNFSM5ZTSXCWA. You are receiving this because you were mentioned.Message ID: @.**@.>>

vijaygit02 commented 2 years ago

@giugent , the enhancement has been added to the master repo and even the document is updated. Thanks again for your involvement in making MDCS better. I'll be closing all the related issues. Commit :https://github.com/Esri/mdcs-py/commit/ee9719c48a8d6116a58da9054dccbceb144dfc51

Best Regards, Vijay

giugent commented 2 years ago

👍

Da: Vijay Pawar @.> Inviato: giovedì 14 luglio 2022 08:31 A: Esri/mdcs-py @.> Cc: Giuseppe Gentili @.>; Mention @.> Oggetto: Re: [Esri/mdcs-py] Import Field Value command (Issue #96)

@giugenthttps://github.com/giugent , the enhancement has been added to the master repo and even the document is updated. Thanks again for your involvement in making MDCS better. I'll be closing all the related issues.

Best Regards, Vijay

— Reply to this email directly, view it on GitHubhttps://github.com/Esri/mdcs-py/issues/96#issuecomment-1184049561, or unsubscribehttps://github.com/notifications/unsubscribe-auth/APMKMUSFPZUSXJ3U6JAIJMTVT6X2LANCNFSM5ZTSXCWA. You are receiving this because you were mentioned.Message ID: @.**@.>>