Having problems self hosting

Tradeforlife commented 4 months ago

Hi,

I've followed the instructions regarding the self hosting, however, I can't seem to get it to work.

I can get the docker working. I can get the git cloned I can create a virtualenv and run the python3 code/main.pu --suburb --state but I'm not sure how I can get the website up and running using my local data.

Tradeforlife commented 4 months ago

I found the docker file and attempted to use that and I get the following error.

gavin@nbnlookup:~/nbn-upgrade-map$ sudo docker-compose -f extra/docker/docker-compose.yaml --profile test up Starting docker_db_1 ... done Starting docker_app_1 ... done Attaching to docker_db_1, docker_app_1 db_1 | db_1 | PostgreSQL Database directory appears to contain a database; Skipping initialization db_1 | db_1 | 2024-06-21 02:45:40.372 UTC [1] LOG: starting PostgreSQL 16.3 on x86_64-pc-linux-musl, compiled by gcc (Alpine 13.2.1_git20240309) 13.2.1 20240309, 64-bit db_1 | 2024-06-21 02:45:40.372 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432 db_1 | 2024-06-21 02:45:40.372 UTC [1] LOG: listening on IPv6 address "::", port 5432 db_1 | 2024-06-21 02:45:40.377 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432" db_1 | 2024-06-21 02:45:40.382 UTC [23] LOG: database system was shut down at 2024-06-21 02:45:36 UTC db_1 | 2024-06-21 02:45:40.388 UTC [1] LOG: database system is ready to accept connections app_1 | 2024-06-21 02:45:40,566 INFO MainThread Checking for externally updated geojson results... app_1 | 2024-06-21 02:45:41,606 INFO MainThread ...done app_1 | 2024-06-21 02:45:41,641 INFO MainThread Checking for unprocessed suburbs... app_1 | 2024-06-21 02:45:41,641 INFO MainThread Checking for all suburbs... app_1 | 2024-06-21 02:45:41,649 INFO MainThread Processing Bolaro, NSW app_1 | 2024-06-21 02:45:41,649 INFO MainThread Fetching all addresses for Bolaro, NSW app_1 | 2024-06-21 02:45:41,649 INFO MainThread Executing query: app_1 | SELECT gnaf_pid, address, postcode, latitude, longitude app_1 | FROM address_principals app_1 | WHERE locality_name = ? AND state = ? app_1 | LIMIT 100000 app_1 | Traceback (most recent call last): app_1 | File "/app/./main.py", line 238, in app_1 | main() app_1 | File "/app/./main.py", line 230, in main app_1 | process_suburb(db, state, suburb, args.threads, progress_bar=args.progress) app_1 | File "/app/./main.py", line 166, in process_suburb app_1 | db_addresses = db.get_addresses(suburb, state) app_1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ app_1 | File "/app/db.py", line 43, in get_addresses app_1 | for row in self.db.execute(query, (target_suburb, target_state)) app_1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ app_1 | File "/app/db.py", line 155, in execute app_1 | self.cur.execute(query, args) app_1 | sqlite3.OperationalError: no such table: address_principals docker_app_1 exited with code 1

LukePrior commented 4 months ago

The running locally guide is probably outdated, will look into this when I get a chance.

Otherwise @lyricnz might know if this is something obvious.

lyricnz commented 4 months ago

I don't use docker Postgresql anymore (too much memory for my little workstation) - I use the sqlite version.

https://github.com/LukePrior/nbn-upgrade-map/releases/tag/sqlite-db-202402

❯ ./code/main.py -H ./address_principals.sqlite
2024-06-22 14:56:25,607 INFO MainThread Checking for externally updated geojson results...
2024-06-22 14:56:26,705 INFO MainThread ...done
2024-06-22 14:56:26,746 INFO MainThread Checking for unprocessed suburbs...
2024-06-22 14:56:26,747 INFO MainThread Checking for all suburbs...
2024-06-22 14:56:26,754 INFO MainThread Processing Cooloola Cove, QLD
2024-06-22 14:56:26,755 INFO MainThread Fetching all addresses for Cooloola Cove, QLD
2024-06-22 14:56:26,755 INFO MainThread Executing query:
            SELECT gnaf_pid, address, postcode, latitude, longitude
            FROM address_principals
            WHERE locality_name = ? AND state = ?
            LIMIT 100000
2024-06-22 14:56:27,649 INFO MainThread Fetched 1944 addresses from database
2024-06-22 14:56:27,656 INFO MainThread Loaded 1709 addresses from output file
2024-06-22 14:56:27,656 INFO MainThread Submitting 1944 requests to add NBNco data...
2024-06-22 14:56:28,243 WARNING nbn_8 No valid suggestions for 7 EXETER COURT COOLOOLA COVE 4580
2024-06-22 14:56:28,547 WARNING nbn_5 No valid suggestions for 36 CANBERRA AVENUE COOLOOLA COVE 4580
...

lyricnz commented 4 months ago

Regarding OP question, they say "how I can get the website up and running using my local data" But that doesn't seem to be the error they pasted. Which is your actual question @Tradeforlife ?

The live website doesn't use the database at all - only the files in /results

To run the website locally is a wee bit of a hassle, due to some hard-coding, and CORS security stuff in your browser.

TLDR:

patch site/main.js to (at least):
- disable the service worker at the top
- change like 247 to url = "../results/" + state_file + ".geojson"

start chrome with a temporary profile, and with CORS/security disabled. On Mac it's something like

mkdir xxx
/Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome --disable-web-security --user-data-dir=$PWD/xxx

open the site URL, and select your suburb
it still uses some of the life data, for list of suburbs, historical data etc. To fix those:
- replace each instance of https://raw.githubusercontent.com/LukePrior/nbn-upgrade-map/main with ..
- comment out the whole block from 413-429

Sample local run

lyricnz commented 4 months ago

@Tradeforlife ?

Tradeforlife commented 4 months ago

sorry, filed this in the too hard basket.

lyricnz commented 4 months ago

Maybe if you let me/us know what you're trying to do, we could make it not-hard?

lyricnz commented 4 months ago

I just wrote a script to generate a local-only (except map tiles) version of the website https://github.com/LukePrior/nbn-upgrade-map/pull/381

With that code:

./code/adhoc_tools.py generate_local_website
python -m http.server 8000
open http://localhost:8000/site/index-local.html

lyricnz commented 4 months ago

@Tradeforlife the code above is merged.

Tradeforlife commented 3 months ago

what about talking to local sites suburb data?

lyricnz commented 3 months ago

@Tradeforlife what do you mean? That's what the instructions above do - they make a patched version of the website which works with whatever data is in your local checkout, then start a little webserver to expose it. Your browser never touches the github.com website for the data, see the details in #381

LukePrior commented 2 months ago

Closing due to lack of activity, @Tradeforlife if you have additional questions please reopen.

LukePrior / nbn-upgrade-map

Having problems self hosting #378