Modify app to output to shape files, instead of direct insert to ArcSDE

[barbeau]

ArcSDE setup is prone to tons of permissions issues, and requires tedious setup, both in terms of the machine running the desktop application as well as the SDE server. If our desktop application was updated to output data directly to OGC-compliant shape files instead, it would allow FDOT to leverage existing processes they have to batch update their system from shape files.

My notes so far:

From a quick review of the Java code on Github, it looks like the code to download the GTFS zip files, extract this data, and store it in a Java object data model is all abstracted from the underlying data store (i.e., ArcSDE Java objects). In other words, you should be able to re-direct the data output function that loops through the GTFS data and writes the data from inserting into SDE into writing to shp files without too much effort.

The tool can be configured to retrieve multiple agencies at once. Here’s the method that loops through each agency’s GTFS data (if you’re only retrieving one agency’s data, this loop executes only once): https://github.com/CUTR-at-USF/RegionalTransitArchitecture/blob/master/Desktop_App/Source_Code/edu/usf/cutr/fdot7/main/Test.java#L283

For each agency, this method is then called, which then loops through the GTFS tables and records in each table and inserts each to SDE: https://github.com/CUTR-at-USF/RegionalTransitArchitecture/blob/master/Desktop_App/Source_Code/edu/usf/cutr/fdot7/main/Test.java#L294

This is where the SDE-specific Java objects are used – the SDE Java library then handles all the mechanics of actually inserting the data into the database. This is the method you’d change to write to shp file(s). It’s up to you how to structure this – you could use one shape file per agency.

From some quick Googling, it looks like there are a number of Java libraries out there than can write to the shp file format – this looks most promising: • GeoTools – I’ve used this before, but not for writing shp files – You’d replace CSV data source in this tutorial with the above GTFS data - http://docs.geotools.org/stable/tutorials/feature/csv2shp.html

From this post, it looks like you could even use GeoTools to directly insert into database, if you wanted: • http://gis.stackexchange.com/questions/154003/how-to-programmatically-import-a-shapefile-into-oracle-db-using-java

For initial database prep - there is a tool to generate the ESRI database schema – you’d either have to get this running, or just generate the tables manually (or via another script). This Java app reads the schema from a config file, which is here: https://github.com/CUTR-at-USF/RegionalTransitArchitecture/blob/master/Desktop_App/Source_Code/data-source.xml#L11

…so you can use this same info to generate the tables/fields manually.

[barbeau]

@knitzman I'm going to keep my notes for what needs to happen to output directly to shape files in this Github issue. So, you may want to follow it.

[barbeau]

It looks like GeoTools actually provides a drop-in replacement for the ArcSDE Java objects SDK, so we should be able to get the project to compile using GeoTools without having to install ArcSDE on a machine: https://github.com/geotools/geotools/tree/master/modules/plugin/arcsde/sde-dummy

[barbeau]

Here's the GeoTools documentation for their ArcSDE plugin: http://docs.geotools.org/latest/userguide/library/data/arcsde.html

[barbeau]

Someone trying to use the ArcSDE Java API to write directly to a shape file: http://forums.esri.com/Thread.asp?c=2&f=59&t=103975

Unfortunately they didn't seem to get it working.

[barbeau]

Some info on converting from ArcSDE Java objects to GeoTools Java objects, and writing to shp file: http://forums.esri.com/Thread.asp?c=2&f=59&t=100178#280482

[barbeau]

Options to implement:

1. Update ArcSDE Java API JAR files to the current version used by FDOT - it might work out of the box and insert directly into ArcSDE
2. Dump data to a local ArcSDE instance and use sde2shp to convert to shape files
3. Dump data to a local geodatabase (not ArcSDE) and export to shape file from there
4. Figure out a way to write directly to a shape file using the ArcSDE Java APIs (doesn't seem possible)
5. Re-write application to create GeoTools Java table/layer objects, and use GeoTools to write directly to shape file - cleanest solution