Selecting between Available .csv files

[SteveCossy]

If you want to stick with the idea of allowing the user to select from multiple geojson files, here's an idea that might work.

Create an "index" or "toc" json file that includes something like the following:

{
    "currentGeo": {
        "title": "Current GeoJSON",
        "path": "geojson/current.geojson"
    },
    "archiveGeo": [
        {
            "title": "Prevous GeoJSON 1",
            "path": "geojson/date-prev1.geojson"
        },
        {
            "title": "Prevous GeoJSON 2",
            "path": "geojson/date-prev2.geojson"
        }
    ]  
}

(Note, the above is just an example)

index.html could then load this json first, render the "currentGeo", and display a list of the "archiveGeo" entries.

Your python script could create and maintain such a file.

Originally posted by @shermp in https://github.com/SteveCossy/IOT/issues/2#issuecomment-559597381

[SteveCossy]

@shermp How tricky is it for Javascript or other technique to trigger a Python app on demand? Doing it that way would greatly reduce (x7 in the current example) the archive storage required. If it is tricky, storage is cheap - we could store the geojson.

Another request - could we have an option of passing the name of a file as a parameter to the URL in the geojson folder that it would use instead of the default?

:-) (Edit - made the other request clearer)

cosste@students:~$ ls -l /var/www/html/OSM/
-rw-rw-r-- 1 cosste cosste 14574 Dec  6 15:50 RSSILatLong.geojson
cosste@students:~$ ls -l CayMQTT/RSSILatLong.csv
-rw-rw-r-- 1 cosste cosste 2106 Dec  6 15:51 CayMQTT/RSSILatLong.csv
cosste@students:~$

[shermp]

This is starting to get into the realm of having a web server application serve the data. PHP is of course the traditional method of doing this (which I don't know), but one can also write a python web app, or Node.JS, or Go, or...

I've used Flask for a previous project. Great for a simple site/page, but starts getting a bit more involved if you need authentication support, or other more complex requirements.

And at that point, one may as well store the data in a database, where the client can do stuff like querying a date range, or other filtering and sorting.

[SteveCossy]

I think I almost have this working, but not quite! It is using the web.py framework. http://webpy.org/docs/0.3/tutorial form2.py is supposed to:

1. Prompt for user to select one file from a list of csv files provided (from ./archives.old at present)
2. run csv2json.py which generates a geojson file from the csv, in the path provided
3. Open a web browser with http://students.pcsupport.ac.nz/OSM/?+name of geojson file just created

Parts 1 and 2 work fine. The script then crashes with a malformed header. Even is a simple test I've been unable to get part 3 working. I would be happy if I could just display the URL for the user to copy and paste into the URL bar. https://github.com/SteveCossy/IOT/commit/79f51dbdf7095c818144775d8f373527030ed3c4

Is this something you'd have desire and time to look at please @shermp ?

[shermp]

First, I've never used web.py, so I don't have any experience with it. The last python based web development I did (a couple of years ago now), I used the Flask framework. However, lets see if I can help figure this out.

Which script actually crashes?

[SteveCossy]

If I comment out the call to csv2json.py, then form2.py works fine (which is pretty much what form.py is). When form2.py calls csv2json.py, then csv2json.py creates the geojson file as it is supposed to (which is the end of this script), but form2.py crashes with the error messages that are in https://github.com/SteveCossy/IOT/blob/master/webpy/error.log One thing I am confident of is that this will be a simple solution, once I find it! Perhaps csv2json.py is not exiting cleanly. I've very curious of the significance of this part of the error message: Getting ./archive.old/RSSILatL The filename is truncated. I tried changing the path (./archive is a symbolic link to a more sensible place) to no avail. Any suggestions welcome and I can make some time to try things, although I'm concentrating on clearing things up and getting ready for the trip to South Taranaki, probably on Sunday. Cheers!

[shermp]

A few things I've spotted.

if (ChosenFile.endswith('.csv')) :
                OutputFile = os.path.join(OutputPath,ChosenFile[:-4])

should probably be something like

fileName, fileExt = os.path.splitext(ChosenFile)
if 'csv' in fileExt:
                ChosenFile = os.path.join(OutputPath, fileName)

The filepath returned needs to be URL encoded (replace space with %20, / with %2F etc). The path also needs to be accessible to the web browser. In this case, that looks the path should be something like temp%2Fgeofilename.geojson. Python has a library for URL encoding a string. Although it seems there's python 2/3 shenanigans going on there...

Also, instead return '\r\nChosenFile', you might want to look at returning a temp redirect to the map, eg:

raise web.seeother('http://students.pcsupport.ac.nz/OSM/?{}'.format(ChosenFile))

[SteveCossy]

That's a few things to work on! I did find a Python library function that can turn a string into a 'URL compatible' form as you suggest. The return string was an experiment, as most posts regarding malformed header seemed to suggest missing carriage return line feeds! :-)

[SteveCossy]

@shermp I've completed most of what you suggested. Exceptions:

• URL did strange things when I used urllib. Code works without it. (form.py:55ff)
• You suggested a more elegant way to do 'nbsp;nbsp;nbsp;|nbsp;nbsp;nbsp;' (index.html:21) but I have lost your suggestion :-(
• The way I have used raise web.seeother( is not at all elegant - it jumps out of an if statement before returning a value from the function. That can't be right! (form.py:57)
• My primitive attempts at Javascript work, but look ugly. Suggestions welcome if you have time.
• Speaking of which, would you like an acknowledgement on the web page? (You did all the important stuff - I just screwed with the appearance!) https://github.com/SteveCossy/IOT/commit/1ca438af741972595cc203a394268a1aa005c4ae Edit - forgot to mention that the header error I was getting resulted from print statements in the Python code I was calling. Commented them out got on with the Real Work!

[shermp]

@shermp I've completed most of what you suggested. Exceptions:

• URL did strange things when I used urllib. Code works without it. (form.py:55ff)
• You suggested a more elegant way to do 'nbsp;nbsp;nbsp;|nbsp;nbsp;nbsp;' (index.html:21) but I have lost your suggestion :-(

Yeah, something like <span class="separator">|</span>, with the appropriate CSS (like .separator {margin: 0 3em;}

• The way I have used raise web.seeother( is not at all elegant - it jumps out of an if statement before returning a value from the function. That can't be right! (form.py:57)

Yeah, that is right. You are essentially raising an exception. Treat it like a return. Goodness knows why the web.py developer decided to handle it that way but...

• Speaking of which, would you like an acknowledgement on the web page? (You did all the important stuff - I just screwed with the appearance!) 1ca438a

It's fine as-is. Thanks for the thought though.

Edit - forgot to mention that the header error I was getting resulted from print statements in the Python code I was calling. Commented them out got on with the Real Work!

Good to hear

[SteveCossy]

Oh dear - spent half an hour trying to find details of how get the spacing to work. Will park that project. Other interesting thing - site doesn't work on my son's desktop (in Chrome or Edge), or on his work phone. Go to the site and get one line 'not found'. Works everywhere else I've tried it. Nothing in the Error Log at the time except many repeats of

Unhandled exception in thread started by
sys.excepthook is missing
lost sys.stderr
Unhandled exception in thread started by
sys.excepthook is missing
lost sys.stderr
Unhandled exception in thread started by
sys.excepthook is missing
lost sys.stderr

Access log has some 404 errors if the URL does not include a trailing /

64.233.173.141 - - [30/Dec/2019:18:13:52 +1300] "GET /OSM/form.py/ HTTP/1.1" 200 659 "-" "Mozilla/5.0 (Linux; Android 8.0.0; F5321) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.93 Mobile Safari/537.36"
64.233.173.139 - - [30/Dec/2019:18:13:53 +1300] "GET /favicon.ico HTTP/1.1" 404 503 "http://students.pcsupport.ac.nz/OSM/form.py/" "Mozilla/5.0 (Linux; Android 8.0.0; F5321) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.93 Mobile Safari/537.36"
101.98.125.252 - - [30/Dec/2019:18:14:10 +1300] "GET /OSM/form.py HTTP/1.1" 404 223 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 Safari/537.36"
101.98.125.252 - - [30/Dec/2019:18:15:02 +1300] "-" 408 0 "-" "-"
101.98.125.252 - - [30/Dec/2019:18:15:10 +1300] "GET /OSM/form.py HTTP/1.1" 404 223 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.18362"
101.98.125.252 - - [30/Dec/2019:18:15:11 +1300] "GET /favicon.ico HTTP/1.1" 404 503 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.18362"
115.189.87.214 - - [30/Dec/2019:18:17:01 +1300] "GET /OSM/form.py HTTP/1.1" 404 223 "-" "Mozilla/5.0 (Linux; Android 9; SM-G390Y) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Mobile Safari/537.36"
115.189.87.214 - - [30/Dec/2019:18:17:01 +1300] "GET /favicon.ico HTTP/1.1" 404 502 "http://students.pcsupport.ac.nz/OSM/form.py" "Mozilla/5.0 (Linux; Android 9; SM-G390Y) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Mobile Safari/537.36"
115.189.87.214 - - [30/Dec/2019:18:17:52 +1300] "-" 408 0 "-" "-"

Not a priority for now ...

EDITED by @shermp improve appearance/readability of logs

[shermp]

I've edited your post to make the logs easier to read/parse, by using code blocks.

For inline blocks you want to style, use single backticks around the snippet you wish to style, like this.

For large blocks, use triple backticks on their own line before and after the block, so it appears like:

This
And this

[shermp]

Some issues might be caused by calling the csv2json script externally like you are doing here:

os.system('python3 csv2json.py '+ChosenFile+' '+OutputFile )

You can import functions/classes from other python files and use them directly. So I would recommend putting the main logic of the csv2json.py script in a function that can be called from form.py.

I can do this if you want me to. It won't take long.

[shermp]

And done. I can push it to the repository if you want.

[SteveCossy]

Please do push. We have reached the end of my current Python string. If I let go now, not sure how long it will take to find another place to rest! (Just watched Star Wars - head is full of deep metaphors etc) Will have a good look at your code when I get home. (Currently lunch on Courtney Place.) Guess I really should build this into a student project, but we don't teach Python (big hole in our current degree implementation). That makes it hard to find students who can produce an applied outcome in the three months or so they have.

[shermp]

Ok, pushed.

Haven't actually tested, but I haven't made any logic changes, just a little code reorganization.

[SteveCossy]

Since when has your code needed testing @shermp? OK - seriously, I'll take credit for writing good code in the first place. Yes, turning this whole process into a set of libraries, and cutting down duplicate code, would be high on the list if I were to do a tidyup, but a tidyup is not on the list!
Thanks for this contribution. I tested it online and with both python and python3 from the command line. Passed with flying colours. Tried running the online code with python3, but my brief attempts could not get the web.py libraries to work. Not a biggie IMHO. Thanks again!

[SteveCossy]

Looks good! http://students.pcsupport.ac.nz/OSM/form.py/