search

mccgr / abn_lookup

Code for creating tables containing the ABN's for companies registered with the Australian Business Register on the ABN lookup website (https://abr.business.gov.au/)
5 stars 3 forks source link

Remove psql dependency #20

Closed iangow closed 4 years ago

iangow commented 4 years ago

https://github.com/mccgr/abn_lookup/blob/d90587c13868aa250f403ff52836507c381126bc/get_abn_lookup_data.py#L36-L67

I think it's more robust not to rely on psql as you do in the code above. See here for an alternative approach (of course, this may use psql behind the scenes, but I think it's better not to have to worry about such details).

bdcallen commented 4 years ago

@iangow I'm in the midst of making a version of the program which makes this change. I do have to say, however, that it potentially ends up changing quite a lot in the program, not just the above function. I use the output and error returned from this function to check whether all the required entries have been written successfully to the database, and to print errors if they occur, in either the xslt step or the psql step. Is there a way to get the cursor object used in the wrds function given above to print out the COPY statement (or output in case of error) given after the processing of the output from the xslt process?

bdcallen commented 4 years ago

@iangow So I rearranged the code so that it does not now use the previous psql commands. All sql commands are now done through sqlalchemy, and the previous sql scripts have been subsumed into the main program and then subsequently deleted. I tested the code on a dummy schema abn_lookup_test first (which I will delete soon), then I finally tested with the code writing to the abn_lookup schema, and everything has worked as expected. So I'm closing this.