[Wiesbaden] Data

[jklmnn]

Wiesbaden also has parking data available at https://www.wiesbaden.de/leben-in-wiesbaden/verkehr/verkehrsinfos/parken.php. The raw data seems to be at https://wi.memo-rheinmain.de/wiesbaden/parkliste.phtml?order=carparks. While it looks quite ugly to parse it seems to have all the data we need.

[kiliankoe]

While it looks quite ugly to parse

Lol, this is far better than most others 😄

[jklmnn]

But only with whitespace correction :smile:

[AugustQu]

Hi,

here I am. From what I understand: first I deliver the coordinates of the parking-lots given in the URL above.

So this may look like: static_data["City I"] = {'latitude': "50.08121", 'longitude': "8.23637", 'adresse': "Schwalbacher Str. 41-43"}

Am I correct?

And a question: this parking-lot is closed on sundays/holidays and opened on monday to saturday from 7:00 to 20:00. How do I enter this information?

PS: I've taken this file as an example: https://github.com/Bytespeicher/pyParkingLotEF2Json/blob/master/pyParkingLotEF2Json.py

[AugustQu]

I've seen a lot of py-files. Looking at the code I found:

• create a loop over all parking-lots (14 for WI)
• search for the name (td-Element)
• next is the 'Freie Stellplätze'
• finally: return the name together wit the numbers.

Am I right?

PS: I'm new to python.

[jklmnn]

Hi,

here I am. From what I understand: first I deliver the coordinates of the parking-lots given in the URL above.

So this may look like: static_data["City I"] = {'latitude': "50.08121", 'longitude': "8.23637", 'adresse': "Schwalbacher Str. 41-43"}

Am I correct?

And a question: this parking-lot is closed on sundays/holidays and opened on monday to saturday from 7:00 to 20:00. How do I enter this information?

PS: I've taken this file as an example: https://github.com/Bytespeicher/pyParkingLotEF2Json/blob/master/pyParkingLotEF2Json.py

The project you are referring to seems to be outdated. Also it doesn't use our API.

The files relevant for you reside in park_api/cities. I'd recommend you to take a look at Sample_City.py and Sample_City.geojson. You could copy those files to Wiesbaden.py and Wiesbaden.geojson respectively and then edit them. Wiesbaden should then automatically appear in your local instance.

For parsing we use beautifulsoup4. I'd advise you to do the same since it is easily powerful enough for the task and already a dependency. About the data the relevant table is accessible via the xpath /html/body/table/tbody/tr[3]/td/div/table/tbody and then iterate over all trs except the first one (which is the table header) and get the data from the tds.

Another far simpler, but important and time consuming task would be filling the geojson file which needs to be done by hand but only once. The most important information are the coordinates.

[AugustQu]

Hi,

I'm not only new to Python but also new to Github.

I copied the whole project as a ZIP-file, expanded it and modified the file _Samplecity.geojson adding all the 14 parking-lots of Wiesbaden. Finally I saved it as Wiesbaden.geojson, locally on my PC.

Some questions:

1: in the 'features' of this file coordinates have to be entered. I entered the coordinates of the city town hall. Is this correct?

2: in the 'properties' I have to enter an URL. Here I entered "https://www.wiesbaden.de/leben-in-wiesbaden/verkehr/verkehrsinfos/parken.php". Correct?

Also as a source I entered: "https://wi.memo-rheinmain.de/wiesbaden/parkliste.phtml?order=carparks". Correct?

3: there is one parking lot that has 2 entries for cars. Can this be handled? Or shall I omit the second entry?

[jklmnn]

I'd recommend to check out the GitHub Hello World tutorial and the git tutorial (while GitHub builds upon git, git is a free version control system). There also is a collection of resources that looks like a good start.

As for the general workflow, you should fork this repository, do your changes on your on fork and the create a pull request with your changes for us to review.

1. The data includes a list of features with the first being the city itself and all others being parking lots. The coordinates of the first feature should be either the geographical center of the city or the geographical center of the distribution of parking lots. We usually take the coordinates of large map providers for this.
2. Yes that's both correct. The url field denotes the public accessible location of the data (city homepage, etc) and the source denotes the actual source of the information that the parser will receive.
3. Do you mean entries in how you enter the lot or data entries? In the first case we only use one address and coordinate. In the second case I'm not sure how it would look like.

[AugustQu]

Hi,

thanks for the link to 'hello world' on GitHub. I will look at it.

Now I have the file Wiesbaden.geojson ready I will check it again, polish it a bit and then open a Pull Request.

In parallel I will try to work on the file Wiesbaden.py.

[jklmnn]

Thanks! I added a branch wiesbaden so when you open the pull request please select this as the target branch since we only allow finished features on master. When everything works wiesbaden will then be merged into master.

[AugustQu]

playing with Python I got this result: my code:

table=soup.select('table') td = table[2].find_all("td") i=0 while i < len(td): print('name:',td[i+1].text.strip()) print('frei=', int(td[i+2].text.split()[0])) print('total=', int(td[i+2].text.split()[2])) i += 5

the result, first entry only:

('name:', u'City I') ('frei=', 137) ('total=', 165)

hint: I added the html statically for my test.

So I have to add these informations to the data-object and I should be done.

Looking into the Sample_City.py I find these lines:

    data["lots"].append({

"name": lot.name, "free": lot_free, "total": lot_total, "address": lot.address, "coords": lot.coords, "state": state, "lot_type": lot.type, "id": lot.id, "forecast": False,

The first 3 lines are the information given above, the rest comes from the geojson-file looked into via the name. Correct?

PS: I have to check for state and lot.id.

[AugustQu]

I added this to my code:

stand=soup.select('span')
last_updated_date=stand[0].text.strip().split()[1] last_updated_time=stand[0].text.strip().split()[2] print("last updated=", last_updated_date, " ", last_updated_time)

which results in this line: ('last updated=', u'07.04.2019', ' ', u'11:16')

[AugustQu]

I just added the files Wiesbaden.geojson and Wiesbaden.py but I'm not sure that I did it correctly.

In the Python-script I added 4 TODOs which describe lines to look for:

• the handling of last_updated
• the state (twice)
• the lot.id which I do not know

Can you please check these lines and also the whole script. Thanks.

[AugustQu]

one error I made: I loaded the file Sample_city.geojson, replaced the contents with my file (Wiesbaden.geojson) and saved it as Wiesbaden.geojson. Instead I should have created a new file via the button.

Same for the py-file.

Sorry.

[jklmnn]

Sorry for my late reply.

The first 3 lines are the information given above, the rest comes from the geojson-file looked into via the name. Correct?

Yes that's true.

There are some issues about the pull requests. One is they're opened against master which we won't merge until everything works perfectly. But you can easily choose the wiesbaden branch with the dropdown on the left:

The second thing is I'd suggest you only make a single pull request with all your changes. This helps us to directly comment on all parts of your code to help you better. By pushing to the branch of the single pull request it will be updated and we always will see the changes. Furthermore having all changes on a single branch helps us checking it out and testing it more easily.

[AugustQu]

what do I have to do now? Add a new Pull-request against the Wiesbaden-branch?

Or do you handle this situation?

[jklmnn]

Yes, you have to open a new pull request against the wiesbaden branch.

[AugustQu]

OK. I added the files Wiesbaden.py and Wiesbaden.geojson in the cities-directory of the Wiesbaden-branch and did a Pull-request. I hope this is correct now.

Unfortunately I added the file Wiesbaden.py to the wrong directory in the first try so please delete this file. And also it looks like I did overwrite the file _Samplecity.py with the code for Wiesbaden. Can I correct this? Or do you simply reject the 'modifications'?

[jklmnn]

Thanks, that looks better. Since the overwritten Sample_City.py file is in its own commit it doesn't pose a problem as I can omit it when merging. The current process will look as follows: I'll review your code, probably request some changes which you then can push to the pull request branch. When we're at the point that the code is clean and working we'll merge the pull request.

[AugustQu]

I will look at the status of the parking-lot over the weekend, then make some changes to the code.

I've seen that the data on wiesbaden.de showed the status of 2 parking-lots as closed, I saw this last night. I will try to find out how they do it.

[AugustQu]

it looks like there are only minor modifications needed. So I will look for the status, make the needed changes plus the already mentioned modifications and deliver them together.

I gues the next version will come on monday.

One hint: the word Uhr is included in the following line taken from the file Sample_city.py:

    "last_updated": convert_date(last_updated, "%d.%m.%Y %H:%M Uhr"),

[jklmnn]

One hint: the word Uhr is included in the following line taken from the file Sample_city.py

This is of course just an example and needs to be adapted to the actual data.

[AugustQu]

OK, I got it.

Uploading my code to the branch wiesbaden I did encounter 2 problems:

• I do not see the old files any more

• so I tried to create new files but could not save them because I have no write access

What can I do now?

[AugustQu]

there is this: AugustQu/ParkAPI there is a wiesbaden-branch where I can find my files.

I will add my latest modifications to these files and then make another pull-request.

[AugustQu]

so I hope I did it correctly.

[AugustQu]

I made some modifications to the py-script and will upload this file later. These modifications were of the type beautifying.

But I do see one problem: there is a parking-lot with the name Coulinstraße. On the memo-rheinmain-page it appears as: Coulinstraße, in the geojson-file I added it as Coulinstrasse. In the py-script this will be handled as: parking_name = td[i+1].text.strip() lot = geodata.lot(parking_name) so the name is extracted from the memo-rheinmain-page and this is used for the search in the geojson-file. This will result in no match.

How can I handle this? Enter the name in the geojson-file as Coulinstraße?

[AugustQu]

I found a typing-error: in the geojson-file the name of a parking-lot is written as: Rhein Main in the memo-rheinmain-site it is written as: RheinMain

I will fix this.

Put this aside: does the script work?

[jklmnn]

Sorry for my late reply, I'm quite busy at the moment.

so I hope I did it correctly. It seems you pushed your changes to the already existing pull request, so yes that's correct. I will review it when I find time.

About ß in street names, they should be usually encoded in UTF-8 that supports those characters. Although if the page doesn't have the correct encoding they will be erroneous. I think we have a solution for this to at least convert those characters to ASCII (@kiliankoe do you remember how we fixed that in other cities?).

Put this aside: does the script work?

Same as above, I can tell when I find time to review (and test ofc.).

[AugustQu]

Sorry for my late reply, I'm quite busy at the moment.

OK. The usual thing: I created some lines and want to see the result immediately.....

I've jut added some minor modifications to the two files. One ist the name 'Coulinstraße' in the geojson-file and the other is a bit of cleaning the code in the py-script.

[jklmnn]

Thanks for the changes, I reviewed them. There are two stray lines in the geojson that shouldn't be there. If you remove them I think it's ready to be merged.

One last thing. There is the active_support flag in the json which decides if the city is enabled by default in the apps. If you want the city to be enabled by default I'd suggest you set it to true.

If the website changes and the parser breaks (happens usually rarely) we will either try to fix it or (if the fix requires more work) we probably set active_support to false again unless someone fixes it. That what the flag is for.

[kiliankoe]

I think we have a solution for this to at least convert those characters to ASCII (@kiliankoe do you remember how we fixed that in other cities?).

We have util.remove_special_chars, but that doesn't appear to be sophisticated enough for this specific use-case. There must be a default and sane way to deal with this, probably even built-in to Python.

[AugustQu]

OK. I've created another pull-request.

I set active_support to true. And I also removed a lot of lines (245 in my copy, 440 here in Github), a copy-paste-error.

Open topic: the sz = ß. It's somehow coded on the web-site. If you can convert this to another (ASCII-)character then I have to change the geojson-file accordingly. I will do that, please let me know.

[AugustQu]

regarding the sz-problem: can you please simply do a test. Maybe ist's already working without any problems.

I just did fetch the page via wget and looked into it. The name of the problematic parking-lot looked like: Coulinstraße, so this would be OK.

If I look at it via my browser here I see it as Coulinstraße So, maybe, it's a problem with the settings of my browser.

[jklmnn]

There is no need to create a new pull request each time. Pushing to the already open one is sufficient. I closed the second pull request as the original one contains all the changes we need.

I took a look at the HTML. The data itself seems to be encoded in UTF-8 correctly but the browser doesn't seem to know. Adding a <meta charset="utf-8"/> manually at the top of the HTML fixes the issue in the browser. Since we want to have things encoded in UTF-8 there's no need to change anything here. @kiliankoe we should probably check how encoding is handled in general in our JSON and if and how we could add encoding information.

From your side there's nothing to do about this issue.

[AugustQu]

OK. I will now wait for the next step.

[jklmnn]

I merged your commits to our wiesbaden branch. Once @kiliankoe reviewed the changes and merged them to master the pull request and this issue will be automatically closed. I can then update the server and Wiesbaden will be available in the app.

[AugustQu]

Thanks!

Can't wait to see it 'live'.

PS: this script was my first work in Python......

[jklmnn]

Wiesbaden is live now: https://parkendd.de/map.html#Wiesbaden Please take a short look if everything is correct, especially coordinates.

If you have a Twitter account I would mention you when I tweet about this.

[AugustQu]

Hi,

THANKS! Looks good.

A short test showed up one problem: Coulinstraße. The sz-problem. And with the name fetched from the web-page there is no match in the geojson-file, so no coordinates are availabel.

#1: what can we do to solve this? Is there a general solution?

#2: I can do a quick-fix, an ugly one. Something linke this: if 'Coulinstr' in parking_name then parking_name='Couloinstraße' or so.

Please let me know.

Regarding publication: I will write a text in my blog presenting this app. And maybe another text about the 'making of' of this app. For this I need your help.

[AugustQu]

OK, changed the code (and added some comments). It seems to work here on my site.

What can I do now? Modify the file Wiesbaden.py in my branch (AugustQu/ParkApi) and start a new pull-request?

[jklmnn]

I'll reopen this issue until this problem is fixed.

Thanks for the fix. Please update your fork since we've also pushed some changes and then create a new pull request with the fix.

[AugustQu]

Done.

[jklmnn]

I've seen you updated your master branch but not your wiesbaden branch. So if you want to do a clean pull request from your wiesbaden branch against our wiebaden branch you have to do the same with the wiesbaden branches as you did with master. But I forgot to tell this in my previous comment, so no worries.

This time I can manually merge the changes as they're small so you don't need to do anything.

[jklmnn]

And thanks for your help!

[AugustQu]

I've seen you updated your master branch but not your wiesbaden branch.

I looked into my branch and do not see a file Wiesbaden.py in my master branch, but I do see it in my wiesbaden-branch.

And the pull-request is made from AugustQu/wiesbaden against offenesDresden/wiesbaden.

Can you please check this again.

[AugustQu]

regarding whitespace:

my editor here does not show this. And is your system so sensible?

[AugustQu]

whitespace: in my file here there is no whitespace at the end of that line. So (again) it looks like a copy-paste-problem.

[jklmnn]

Trailing whitespace doesn't necessarily break anything on the running server. But it is at least unclean code which we want to prevent as much as possible and it may break data that spans over multiple lines if ignored. Related: https://softwareengineering.stackexchange.com/a/121560

Also it generates changes in files that are no longer easily visible creating different versions of the same software that have no visible change. Example: In this case a trailing whitespace has been introduced but for the reader it is not clear what has changed. Also there are editor(-plugins) that show trailing whitespace. When I open this file in Vim the whitespace is marked in red:

[AugustQu]

But it is at least unclean code

OK.

I have used gedit for editing the script but I can switch to eclipse which shows whitespace.

Nevertheless this whitespace was entered by copying the lines from my file to the file here.....

[AugustQu]

3 simple questions:

1: in the app I've seen lines in green, yellow, red and blue.

green lines mean a lot of free space - OK. yellow lines mean percentage has fallen under a value of ?? (which value is it?) red lines mean only some parking spots are available (from which percentage on the color will change?) blue line means ?????

2: I've seen texts like 'no data availble'. Will this text appear when my code sets parking_state = 'nodata'?

3: closed parking-lots disappear from the list. Is this the intended behaviour?

[jklmnn]

I assume you talk about the Android app which I wrote. I can't tell you about the iOS and Windows apps.

1. green: 100% free - 20% free yellow: 20% free - 5% free red: <5% free, closed blue: no data available
2. yes
3. If you tap the three dots in the upper right corner and select settings you get to the settings dialog. There you can choose which parking lots are shown. By default full (0% free) and closed lots are hidden.