scraper not working?

[oz123]

Hi Tom,

I suspect the scraper is still not working. After I ran the scraper the entries in my DB look like this:

In [45]: Record.objects.filter(station=Station.objects.get(url_id=7))
Out[45]: 
[<Record: Parameter: Temp.
Station url_id: 7.
Timestamp: 2013-12-23 19:00.
Value: 12.8.>, <Record: Parameter: PM10.
Station url_id: 7.
Timestamp: 2013-12-23 19:00.
Value: 32.0.>, <Record: Parameter: No.
Station url_id: 7.
Timestamp: 2013-12-23 19:00.
Value: 2.1.>, <Record: Parameter: Rain.
Station url_id: 7.
Timestamp: 2013-12-23 19:00.
Value: 0.0.>, <Record: Parameter: SO2.
Station url_id: 7.
Timestamp: 2013-12-23 19:00.
Value: 0.5.>, <Record: Parameter: Nox.
Station url_id: 7.
Timestamp: 2013-12-23 19:00.
Value: 20.4.>, <Record: Parameter: O3.
Station url_id: 7.
Timestamp: 2013-12-23 19:00.
Value: 26.0.>, <Record: Parameter: No2.
Station url_id: 7.
Timestamp: 2013-12-23 19:00.
Value: 18.2.>, <Record: Parameter: RH.
Station url_id: 7.
Timestamp: 2013-12-23 19:00.
Value: 43.0.>]

In [46]: Record.objects.filter(station=Station.objects.get(url_id=7))
Out[46]: 
[<Record: Parameter: Temp.
Station url_id: 7.
Timestamp: 2013-12-23 19:00.
Value: 12.8.>, <Record: Parameter: PM10.
Station url_id: 7.
Timestamp: 2013-12-23 19:00.
Value: 32.0.>, <Record: Parameter: No.
Station url_id: 7.
Timestamp: 2013-12-23 19:00.
Value: 2.1.>, <Record: Parameter: Rain.
Station url_id: 7.
Timestamp: 2013-12-23 19:00.
Value: 0.0.>, <Record: Parameter: SO2.
Station url_id: 7.
Timestamp: 2013-12-23 19:00.
Value: 0.5.>, <Record: Parameter: Nox.
Station url_id: 7.
Timestamp: 2013-12-23 19:00.
Value: 20.4.>, <Record: Parameter: O3.
Station url_id: 7.
Timestamp: 2013-12-23 19:00.
Value: 26.0.>, <Record: Parameter: No2.
Station url_id: 7.
Timestamp: 2013-12-23 19:00.
Value: 18.2.>, <Record: Parameter: RH.
Station url_id: 7.
Timestamp: 2013-12-23 19:00.
Value: 43.0.>]

The Timestamp of Record is the time I ran the script ... I suspect also that the Model concept is quite tricky. Maybe we should discuss this via Skype? I see also you are working on the Parameter view, what is this view supposed to do?

Can you please guide me through how the scraper is supposed to work and what is the expected output?

[shaief]

Hi Oz, After migrating the DB, try running: ./manage.py scrape_stations_info This should create all the stations info in the DB. I'm not sure what happened with the time stamp, though the Timestamp should be somehow close to the time of running the scraper, because the stations suppose to upload data every five minutes.

The parameter view is something Tom and i worked on today, while creating the parameter template as well. Pull the repository and then go to http://localhost:8000/parameter/[paramete name]. For example, go to: http://localhost:8000/parameter/Temp/ http://localhost:8000/parameter/SO2/ http://localhost:8000/parameter/CO/

Hope that helps,

Shai.

[Nagasaki45]

Hi Oz,

It seems that the data is fine. You can inspect it better in the admin. I will add a note regarding scrape_station_info in the readme tommorow.

[Nagasaki45]

@oz123 I've checked the scraper again. I do found a bug, and fixed it, but it have nothing to do with what you asked. As Shai said, the stations themselves update new records every 5 minutes or so (at least most of them). I'm closing the issue for now, you are welcome to open it again if you find any more bugs.