Because it is unknown when IBTrACS data will be updated we must poll the files (currently being done via rundeck using wget calls with -N option).
Using the -N option wget will only fetch files when they are newer on the destination server than they are on the DEV server. When a new file is downloaded we should have a process execute to push the data into PostgreSQL but not when the data hasn't changed.
Integration script should take an md5 hash or keep track of last modified datetime/size to indicate when to clear and replace the PostgreSQL data.
Because it is unknown when IBTrACS data will be updated we must poll the files (currently being done via rundeck using wget calls with -N option).
Using the -N option wget will only fetch files when they are newer on the destination server than they are on the DEV server. When a new file is downloaded we should have a process execute to push the data into PostgreSQL but not when the data hasn't changed.
Integration script should take an md5 hash or keep track of last modified datetime/size to indicate when to clear and replace the PostgreSQL data.