NSW-OEH-EMS-KST / grid-garage

ArcGIS python toolbox supporting table-based processing
4 stars 2 forks source link

New Tool - Join Table (Aligns with Tabulate Intersection) #232

Open imagesbyjohnyoung opened 5 years ago

imagesbyjohnyoung commented 5 years ago

@TomWaterat @byezy The purpose of this tool is to join multiple tables (generated from e.g. Tabulate Intersection process) to a base zone/grid file.

Ideally, in a search table (this having a field that has the geodata, the table name, and field to join on, and priority order) it could be used to loop through, based on priority (important to be able to group like features together) and apply the joins to the zone/grid file.

Currently there appears to be an issue with the standard join table tool (Data Mgmt > Joins > Join Field) with being slow to process. This may be due to no indexing.
E.g. of slowness 17hrs to join ~45 tables to a hex grid of 28k features.

It does appear as a common question to esri.

Attached is an alternative code from Esri, that seems quicker, but having issues when attempting to join multiple tables as a batch.

The ability of joining tables (even though no data may exist after tabulate intersect) still needs to be available. Reason, this identifies that the particular layer/asset doesn't exist within the zones - important to know this. Join_Field.zip

byezy commented 5 years ago

Hi John to start this tool we will need some examples of what you are looking to accomplish... i.e. some data in an fgdb and an mxd with inputs and results, push that into it's own branch in the test repo (ask Tom or I if unsure how to do this when you have some data ready). Making a clip of representative data for development makes it easier if the data are large sets. That code from esri is helpful but a little obtuse, the python is not too flash. We should be able to do this using pandas, which will be blisteringly fast in comparison to joining using loops - but need some examples to be sure about this. Looks like fun!

imagesbyjohnyoung commented 5 years ago

Will get to you asap. Not in Cowra. I will try get something tomorrow

On Fri, 5 Jul. 2019, 7:12 pm byezy, notifications@github.com wrote:

Hi John to start this tool we will need some examples of what you are looking to accomplish... i.e. some data in an fgdb and an mxd with inputs and results, push that into it's own branch in the test repo (ask Tom or I if unsure how to do this when you have some data ready). Making a clip of representative data for development makes it easier if the data are large sets. That code from esri is helpful but a little obtuse, the python is not too flash. We should be able to do this using pandas, which will be blisteringly fast in comparison to joining using loops - but need some examples to be sure about this. Looks like fun!

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/NSW-OEH-EMS-KST/grid-garage/issues/232?email_source=notifications&email_token=AEJPBYOVYR2O2DMNKDNN4VLP54F75A5CNFSM4HRNY3W2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZJAE6I#issuecomment-508691065, or mute the thread https://github.com/notifications/unsubscribe-auth/AEJPBYP6MZHLC3Y5QHGFHCTP54F75ANCNFSM4HRNY3WQ .

byezy commented 5 years ago

No rush at all... just making a start.

On Fri, 5 Jul. 2019, 8:43 pm ArcPY Coders, notifications@github.com wrote:

Will get to you asap. Not in Cowra. I will try get something tomorrow

On Fri, 5 Jul. 2019, 7:12 pm byezy, notifications@github.com wrote:

Hi John to start this tool we will need some examples of what you are looking to accomplish... i.e. some data in an fgdb and an mxd with inputs and results, push that into it's own branch in the test repo (ask Tom or I if unsure how to do this when you have some data ready). Making a clip of representative data for development makes it easier if the data are large sets. That code from esri is helpful but a little obtuse, the python is not too flash. We should be able to do this using pandas, which will be blisteringly fast in comparison to joining using loops - but need some examples to be sure about this. Looks like fun!

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub < https://github.com/NSW-OEH-EMS-KST/grid-garage/issues/232?email_source=notifications&email_token=AEJPBYOVYR2O2DMNKDNN4VLP54F75A5CNFSM4HRNY3W2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZJAE6I#issuecomment-508691065 , or mute the thread < https://github.com/notifications/unsubscribe-auth/AEJPBYP6MZHLC3Y5QHGFHCTP54F75ANCNFSM4HRNY3WQ

.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/NSW-OEH-EMS-KST/grid-garage/issues/232?email_source=notifications&email_token=ABEF6HWLX6XNUGBC3F4ERYDP54QT7A5CNFSM4HRNY3W2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZJGTSY#issuecomment-508717515, or mute the thread https://github.com/notifications/unsubscribe-auth/ABEF6HWSYJ4W742QD3CKQZTP54QT7ANCNFSM4HRNY3WQ .