Deployment and test server

jklmnn commented 2 years ago

I'd like to deploy this software on our server for a test run yet I have a few questions:

Is the current database still subject to change? Could we keep the database of the current test run if we decided that the state of the API is ready for production use?
How far away are we from v1 compatibility?

defgsus commented 2 years ago

The database:

Currently i am stacking django migrations on top of each other like here because i also have a test system running. But for release i would like to squash them together. Django does have a squash command for different migrations but it sometimes creates problems. So the clenest way is to throw all migrations away and create a new initial one (and start with a new database).

So steps will be:

merge the current legacy-id branch
replace the collected migration files with a single one

If we have DB changes later we can add new migration files

V1 compatibility:

Well, the v1 API endpoints look exactly the same. The data is a bit different as described in #4 (lot_ids mainly).

I'll setup the master branch and let you know.

defgsus commented 2 years ago

You can have a go.

I've just tested it with a fresh database.

Frankfurt is still offline
my requests/openssl version does not like the current Wiesbaden certificate (validation can be turned off in the pool settings)
the ./manage.py pa_find_locations does not assign the Bundesland Hamburg to all lots (state field)

apart from that, all seems to work

jklmnn commented 2 years ago

Okay the scraper works now! I had a few errors for some cities but I think for testing we can ignore these for the time being. IIRC the ParkAPI also does not check certificates due to requests not accepting some certificates. I ran the scrapers and still didn't get any cities when I run the API. I suppose I have to run ./manage.py pa_find_locations or did I configure something incorrectly?

When running ./manage.py pa_find_locations I encountered the following error:

./manage.py pa_find_locations                                                                                                                                                                                  :(
580 lots to find
NominatimApi: requesting GET https://nominatim.openstreetmap.org/reverse {'params': {'lat': 50.774352, 'lon': 6.099715, 'zoom': 10, 'format': 'geojson', 'addressdetails': 1, 'namedetails': 1, 'extratags': 1, 'polygon_geojson': 0}}
NominatimApi: writing cache '/.../ParkAPI2/web/cache/nominatim/ad6ccac2cfaa8686118c691816fc84dd.json'
Traceback (most recent call last):
  File "/.../ParkAPI2/venv/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedColumn: column locations_location.geo_polygon does not exist
LINE 1: ...osm_id", "locations_location"."geo_point"::bytea, "locations...
                                                             ^

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/.../ParkAPI2/web/./manage.py", line 22, in <module>
    main()
  File "/.../ParkAPI2/web/./manage.py", line 18, in main
    execute_from_command_line(sys.argv)
  File "/.../ParkAPI2/venv/lib/python3.9/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
    utility.execute()
  File "/.../ParkAPI2/venv/lib/python3.9/site-packages/django/core/management/__init__.py", line 413, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/.../ParkAPI2/venv/lib/python3.9/site-packages/django/core/management/base.py", line 354, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/.../ParkAPI2/venv/lib/python3.9/site-packages/django/core/management/base.py", line 398, in execute
    output = self.handle(*args, **options)
  File "/.../ParkAPI2/web/park_api/management/commands/pa_find_locations.py", line 26, in handle
    find_locations(pools=pools, print_to_console=True)
  File "/.../ParkAPI2/web/park_api/management/commands/pa_find_locations.py", line 98, in find_locations
    location_model = create_location_model(
  File "/.../ParkAPI2/web/park_api/management/commands/pa_find_locations.py", line 136, in create_location_model
    location_model = Location.objects.get(osm_id=osm_id)
  File "/.../ParkAPI2/venv/lib/python3.9/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/.../ParkAPI2/venv/lib/python3.9/site-packages/django/db/models/query.py", line 431, in get
    num = len(clone)
  File "/.../ParkAPI2/venv/lib/python3.9/site-packages/django/db/models/query.py", line 262, in __len__
    self._fetch_all()
  File "/.../ParkAPI2/venv/lib/python3.9/site-packages/django/db/models/query.py", line 1324, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
  File "/.../ParkAPI2/venv/lib/python3.9/site-packages/django/db/models/query.py", line 51, in __iter__
    results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
  File "/.../ParkAPI2/venv/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1175, in execute_sql
    cursor.execute(sql, params)
  File "/.../ParkAPI2/venv/lib/python3.9/site-packages/django/db/backends/utils.py", line 66, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/.../ParkAPI2/venv/lib/python3.9/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/.../ParkAPI2/venv/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/.../ParkAPI2/venv/lib/python3.9/site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/.../ParkAPI2/venv/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: column locations_location.geo_polygon does not exist
LINE 1: ...osm_id", "locations_location"."geo_point"::bytea, "locations...
                                                             ^

Do you have any idea?

defgsus commented 2 years ago

Yes, pa_find_locations is required to get the city names and to populate the database for the API V1.

And no! No immediate idea. Except that the migrations did not run correctly, but why?

If you call ./manage.py showmigrations it should look like the following:

admin
 [X] 0001_initial
 [X] 0002_logentry_remove_auto_add
 [X] 0003_logentry_add_action_flag_choices
auth
 [X] 0001_initial
 [X] 0002_alter_permission_name_max_length
 [X] 0003_alter_user_email_max_length
 [X] 0004_alter_user_username_opts
 [X] 0005_alter_user_last_login_null
 [X] 0006_require_contenttypes_0002
 [X] 0007_alter_validators_add_error_messages
 [X] 0008_alter_user_username_max_length
 [X] 0009_alter_user_last_name_max_length
 [X] 0010_alter_group_name_max_length
 [X] 0011_update_proxy_permissions
 [X] 0012_alter_user_first_name_max_length
contenttypes
 [X] 0001_initial
 [X] 0002_remove_content_type_name
locations
 [X] 0001_initial
park_data
 [X] 0001_initial
sessions
 [X] 0001_initial

locations 0001_initial is the one that seems to be missing... Though, the backend would actually complain about the whole table being missing, not an individual field

In sudo -u postgres psql it should look like this:

\c <the-parkapi-db-name>
\d locations_location

     Column     |            Type             | Collation | Nullable |                    Default                     
----------------+-----------------------------+-----------+----------+------------------------------------------------
 id             | bigint                      |           | not null | nextval('locations_location_id_seq'::regclass)
 date_created   | timestamp with time zone    |           | not null | 
 date_updated   | timestamp with time zone    |           | not null | 
 osm_id         | character varying(32)       |           | not null | 
 geo_point      | geometry(Point,4326)        |           |          | 
 geo_polygon    | geometry(MultiPolygon,4326) |           |          | 
 osm_properties | jsonb                       |           |          | 
 city           | character varying(64)       |           |          | 
 state          | character varying(64)       |           |          | 
 country        | character varying(64)       |           |          | 
 country_code   | character varying(2)        |           |          |

If this looks different then we need to dig deeper there.

It's also possible to migrate back and forth in django (if there is no data that would be lost).

./manage.py migrate locations zero
./manage.py migrate

The first line not only removes the locations table but also the park_data tables, because they depend on locations. Second line restores all tables.

EDIT:

Stuff like this usually happens when developer A applies migration XY and developer B replaces migration XY, instead of adding a new one. That's what i did earlier (2428c24) to clean the plate before you apply the migrations on the server. But looking at the commit now i would bet a KolleMate that you applied locations 0001_initial before i replaced it, because the geo_polygon field is actually the line that's changed.

In that case the backward migration mentioned above will likely not work and the easiest way is to DROP DATABASE and start anew. (In case one actually wants to retain data there are tricks and tweaks possible..)

jklmnn commented 2 years ago

Good news! After a long time ParkAPI2 is finally up and running and you can access it on https://api2.parkendd.de/. Once everything is set we can move the pure API to https://api.parkendd.de/ and retire ParkAPI.

defgsus commented 2 years ago

Greetings JK. that sounds great. I was fighting with myself to split the scrapers/data-sources and apply the recent two fixes, ... but, a quite disintegrating covid wave was passing our family household. Nothing serious happened, we where all just trying to manage the different necessities like school, work, sweating and being good parents. Currently, everybody seems to be healthy again and life got easier.

However. That's the things i want to do next. Extracting the data-sources repo and applying the scraper fixes. And adjusting the github CI action.

Have a good week!

jklmnn commented 2 years ago

After the current outage I noticed that the ParkAPI2 is in a much better shape than the ParkAPI, so I think I will do some last compatibility checks and then change the main API to run on ParkAPI2.

defgsus commented 2 years ago

Hi @jklmnn , sounds great. If questions come up, i can help through office hours.. .. Otherwise i enjoy summer without a laptop. It's a new and rewarding experience :guitar:

jklmnn commented 2 years ago

No worries, so far it's running again, enjoy your vacation!

ParkenDD / ParkAPI2

Deployment and test server #5