search

cgs-earth / wis2box

WIS 2.0 in a box
https://docs.wis2box.wis.wmo.int
Apache License 2.0
0 stars 0 forks source link

Oregon Arcgis to STA #3

Open C-Loftus opened 3 weeks ago

C-Loftus commented 2 weeks ago

@webb-ben just finished this. Tried to make very few changes to the other parts of the codebase and isolated changes into the oregon/ folder. I think that is a good pattern. I tried to make the cli very simple. You just need to run wis2box oregon load in the container

C-Loftus commented 2 weeks ago

May also make sense to try and merge any sta work into a central branch at some point so it is easier to add changes without needing to repeat them across branches. (i.e. stuff like the docker volume for reloading or if i get pytest set up)

C-Loftus commented 2 weeks ago

Added a helper code for storing the last download and updating to only add new data. I am just doing this by reading a file stored on disk. I think this is reasonable for the time being. This week it will be useful to align on our future plans for this branch and how it will be in relation to other branches.

I think I will probably need to write some pytest code for this but wanted to be aligned first in case any of this needs to be refactored

C-Loftus commented 1 week ago

Had some weird issues with aiohttp where the async functions were seeming to hang, but just switching to httpx and changing a few things syntax-wise seemed to fix it. Not really entirely clear what was going wrong. Load time down to 2m 15 seconds for one arbitrary station.

Added a debugger which can be ran inside the container by doing stuff like this debugpy --wait-for-client --listen 0.0.0.0:5678 $(which wis2box) oregon test -k "fully"

Going to make the update functionality a bit more robust on monday. So there is a cache on disk where each station is associated with a time and that time is gotten by the tsv instead of what is passed into the CLI. Then once that is tested and stable I will get the crontab working and get it on google cloud.

Once I get that done I think I will look into pagination on the hub2box UI.

I also think there are some things that need to be cleaned up in wis2box, namely using Protocols and structural typing instead of using inheritance. In some cases the child class inherits from the parent in a non-typesafe way which makes it harder to know what is going on. I generally think that inheritance should be avoided if we can't use type checking with it.

Just some thoughts after looking at this more.