Repo Setup

[cloneofghosts]

Here's a list of things that should probably be addressed before the repository becomes public:

• [x] Proper README.md file
• [x] CODEOWNERS file - When you create a PR it will automatically request anyone who is a CODEOWNER for review. I have this added in the docs/HA repo and it assigns you as a reviewer whenever people submit a PR
• [x] CONTRIBUTING.md file
• [ ] Issue Templates
• [ ] Pull Request template - Maybe something similar to what HA has?
• [ ] Issue config file to link to the other repositories
• [ ] Build test action?
• [x] Lint action?
• [x] Dependabot? Renovate?
• [x] CodeQL?

For the issue templates what sort of fields do you think that you'll need and what sort of templates should there be? Bug, feature request and generic? The linking part is just creating a .config file in the ISSUE_TEMPLATE folder.

The lint action I could probably re-use the same one that I added to HA. Not sure how complex adding a build action would be but would be nice to have if possible.

Dependabot/renovate is mainly for keeping dependencies up-to-date and would probably need to be tested to make sure everything works before rolling it out live.

[alexander0042]

Other than that script added to get something there, I love using this structure! However, the more I think about it the more I think we should just put things in the main repository, since it's the public and popular one that people are watching, so more visibility. A lot of things can stay the same, but a few big changes:

1. Super curious what you think, but I think keeping the permissive license makes sense. I'm always a little worried that companies will just take the code instead of supporting the project; however, it's sufficiently resource intensive to run that I'm not too worried about that for the most part. I like the idea of AGPL, but would need some sort of dual license contributor statement, and after the whole redis debacle that feels weird.
2. Setup a docker action to generate the two containers (ingest and response) based on the dockerfiles. I'm not sure if there can be two separate containers generated in one repo, so might have to merge them, but they're pretty similar. The basic approach is that the ingest containers run ingest scripts and save the data to zip files, and the response container syncs those to databases and serves the responses from that.
3. Tests. Tests for this have always been tricky, since it needs so much static data to do anything. No good solution here.
4. Contributors: I love it when I see these on projects, so something explaining that contributions are welcome, especially with respect to optimizing my terrible python code

[cloneofghosts]

Part of my thinking for a separate repository for the code was so that things are separated so the API issues are in one spot and issues with self-hosting are in another. If you'd rather everything be in one repository that works too but we'd need to setup issue templates for the self-hosting stuff and maybe add a label to distinguish between the two?

1. I'm not the best person to ask since I know next to nothing about different licenses. Looking at other repos I follow: Open-Meteo has a AGPL license, Breezy Weather uses LGPL, Mihon (Manga reader app) uses Apache and OneDrive Linux client is GPL.
2. I think this should be possible> I know GitHub actions you can setup multiple steps so I don't see why this wouldn't be possible. You may need to play around and find out?
3. For this I was more thinking of an action which make sure the containers can be built without any errors. People should ideally make sure they test it locally before submitting a PR but it would be good to have anyway.
4. The one nice thing about the CONTRIBUTING.md file is it displays as a link when you submit a PR but will obviously need docs on the site on how to host yourself.

[cloneofghosts]

Was playing around with the icon logic and came up with something like this:

if (((minuteDict[0]['precipIntensity']) => (0.02 * prepIntensityUnit)) & (minuteDict[0]['precipType'] != None)):
        # If more than 25% chance of precip, then the icon for whatever is happening, so long as the icon exists
        cIcon = minuteDict[0]['precipType']

        if minuteDict[0]['precipType'] == 'rain':
            if minuteDict[0]['precipIntensity'] <= 0.4:
                cText = 'Drizzle'
            elif minuteDict[0]['precipIntensity'] > 0.4 and minuteDict[0]['precipIntensity'] <= 2.5:
                cText = 'Light Rain'
            elif minuteDict[0]['precipIntensity'] > 2.5 and minuteDict[0]['precipIntensity'] <= 10:
                cText = 'Rain'
            else:
                cText = 'Heavy Rain'
        elif minuteDict[0]['precipType'] == 'snow':
            if minuteDict[0]['precipIntensity'] <= 0.13:
                cText = 'Flurries'
            elif minuteDict[0]['precipIntensity'] > 0.13 and minuteDict[0]['precipIntensity'] <= 0.83:
                cText = 'Light Snow'
            elif minuteDict[0]['precipIntensity'] > 0.83 and minuteDict[0]['precipIntensity'] <= 3.33:
                cText = 'Snow'
            else:
                cText = 'Heavy Snow'
        elif minuteDict[0]['precipType'] == 'sleet':
            elif minuteDict[0]['precipIntensity'] <= 0.83:
                cText = 'Light Sleet'
            elif minuteDict[0]['precipIntensity'] > 0.83 and minuteDict[0]['precipIntensity'] <= 3.33:
                cText = 'Sleet'
            else:
                cText = 'Heavy Sleet'
        else:
            if minuteDict[0]['precipIntensity'] <= 0.4:
                cText = Very Light Precipitation'
            elif minuteDict[0]['precipIntensity'] > 0.4 and minuteDict[0]['precipIntensity'] <= 2.5:
                cText = 'Light Precipitation'
            elif minuteDict[0]['precipIntensity'] > 2.5 and minuteDict[0]['precipIntensity'] <= 10:
                cText = 'Precipitation'
            else:
                cText = 'Heavy Precipitation'

        # Because soemtimes there's precipitation not no type, don't use an icon in those cases

    # If visibility <1km and during the day
    # elif InterPcurrent[14]<1000 and (InterPcurrent[0]>InterPday[0,16] and InterPcurrent[0]<InterPday[0,17]):
    elif InterPcurrent[14] < (1000 * visUnits):
        cIcon = 'fog'
        cText = 'Foggy'
    elif InterPcurrent[12] > 0.875:
        cIcon = 'cloudy'
        cText = 'Cloudy'
    elif InterPcurrent[12] > 0.375:

        if InterPcurrent[12] > 0.625:
            cText = 'Mostly Cloudy'
        else:
            cText = 'Partly Cloudy'

        if InterPcurrent[0] < InterSday[0, 17]:
            # Before sunrise
            cIcon = 'partly-cloudy-night'
        elif InterPcurrent[0] > InterSday[0, 17] and InterPcurrent[0] < InterSday[0, 18]:
            # After sunrise before sunset
            cIcon = 'partly-cloudy-day'
        elif InterPcurrent[0] > InterSday[0, 18]:
            # After sunset
            cIcon = 'partly-cloudy-night'
    else:
        if InterPcurrent[12] > 0.125:
            cText = 'Mostly Clear'
        else:
            cText = 'Clear'

        if InterPcurrent[0] < InterSday[0, 17]:
            # Before sunrise
            cIcon = 'clear-night'
        elif InterPcurrent[0] > InterSday[0, 17] and InterPcurrent[0] < InterSday[0, 18]:
            # After sunrise before sunset
            cIcon = 'clear-day'
        elif InterPcurrent[0] > InterSday[0, 18]:
            # After sunset
            cIcon = 'clear-night'

# Show the wind icon if there is no precipitation and no fog.
if minuteDict[0]['precipIntensity'] < 0.02 and InterPcurrent[14] >= (1000 * visUnits):
    # Show the wind text before the sky text
    if InterPcurrent[9] > (6.7056 * windUnit):
        cIcon = 'wind'
        cText = 'Breezy and ' + cText
    elif InterPcurrent[9] > (10 * windUnit):
        cIcon = 'wind'
        cText = 'Windy and ' + cText
    elif InterPcurrent[9] > (17.8816 * windUnit):
        cIcon = 'wind'
        cText = 'Dangerously Windy and ' + cText
else:
    # Show the precipitation text or fog text before the wind text
    if InterPcurrent[9] > (6.7056 * windUnit):
        cText += ' and Breezy'
    elif InterPcurrent[9] > (10 * windUnit):
        cText = ' and Windy'
    elif InterPcurrent[9] > (17.8816 * windUnit):
        cText = ' and Dangerously Windy'

I didn't touch the logic for daily and hourly but it should be simple enough to expand the current logic. I did add Mostly Clear to the list of condition texts which Dark Sky didn't have but considering most other sites have it I figured it would be easy enough to add.

I didn't add logic for Dry, Humid, Thunderstorms and any of the Possible X texts. Dry, Humid and the Possible X texts I'm not sure how exactly they would be implemented and Thunderstorms we have no way to calculate that at the moment.

Dangerously Windy I used that NOAA resource I linked in the fog task to setup Breezy and Dangerously Windy. Breezy is set to 15 mph (6.7056 m/s or 24.14016 km/h) and Dangerously Windy is 40 mph (17.8816 m/s or 64.37376 km/h). I think I also have it setup so that if no precipitation/fog it will show the wind text then the sky text. So if Partly Cloudy and Breezy it will say Breezy and Partly Cloudy whereas in the other case its the opposite so Light Rain and Windy.

[cloneofghosts]

Here's what I came up with for the hourly blocks and I'll look at doing the daily blocks later. Daily will be trickier since I'd probably have to use accumulation instead of intensity but it can work.

Are the intensities converted by this point or are they still in mm? If they're converted then the checks will need to be converted.

# Set text
        if InterPhour[idx, 3] >= 0.3 and (((InterPhour[idx, 21] + InterPhour[idx, 23]) => (0.02 * prepAccumUnit)) or (InterPhour[idx, 22] => (0.02 * prepAccumUnit))):
            # If more than 30% chance of precip at any point throughout the day, then the icon for whatever is happening
            # Thresholds set in mm
            hourIcon = pType
            if pType == 'rain':
                if InterPhour[idx, 2] <= 0.4:
                    cText = 'Drizzle'
                elif InterPhour[idx, 2] > 0.4 and InterPhour[idx, 2] <= 2.5:
                    cText = 'Light Rain'
                elif InterPhour[idx, 2] > 2.5 and InterPhour[idx, 2] <= 10:
                    cText = 'Rain'
                else:
                    cText = 'Heavy Rain'
            elif pType == 'snow':
                if InterPhour[idx, 2] <= 0.13:
                    cText = 'Flurries'
                elif InterPhour[idx, 2] > 0.13 and InterPhour[idx, 2] <= 0.83:
                    cText = 'Light Snow'
                elif InterPhour[idx, 2] > 0.83 and InterPhour[idx, 2] <= 3.33:
                    cText = 'Snow'
                else:
                    cText = 'Heavy Snow'
            elif pType == 'sleet':
                elif InterPhour[idx, 2] <= 0.83:
                    cText = 'Light Sleet'
                elif InterPhour[idx, 2] > 0.83 and InterPhour[idx, 2] <= 3.33:
                    cText = 'Sleet'
                else:
                    cText = 'Heavy Sleet'
            else:
                if InterPhour[idx, 2] <= 0.4:
                    cText = Very Light Precipitation'
                elif InterPhour[idx, 2] > 0.4 and InterPhour[idx, 2] <= 2.5:
                    cText = 'Light Precipitation'
                elif InterPhour[idx, 2] > 2.5 and InterPhour[idx, 2] <= 10:
                    cText = 'Precipitation'
                else:
                    cText = 'Heavy Precipitation'

            hourText = pText
        # If visibility <1000 and during the day
        # elif InterPhour[idx,14]<1000 and (hour_array_grib[idx]>InterPday[dCount,16] and hour_array_grib[idx]<InterPday[dCount,17]):
        elif InterPhour[idx, 15] < (1000 * visUnits):
            hourIcon = 'fog'
            hourText = 'Foggy'
        elif InterPhour[idx, 13] > 0.875:
            hourIcon = 'cloudy'
            hourText = 'Cloudy'
        elif InterPhour[idx, 13] > 0.375:
            if InterPhour[idx, 13] > 0.625:
                hourText = 'Mostly Cloudy'
            else:
                hourText = 'Partly Cloudy'

            if hour_array_grib[idx] < InterSday[hourlyDayIndex[idx], 17]:
                # Before sunrise
                hourIcon = 'partly-cloudy-night'
            elif hour_array_grib[idx] >= InterSday[hourlyDayIndex[idx], 17] and hour_array_grib[idx] <= InterSday[
                hourlyDayIndex[idx], 18]:
                # After sunrise before sunset
                hourIcon = 'partly-cloudy-day'
            elif hour_array_grib[idx] > InterSday[hourlyDayIndex[idx], 18]:
                # After sunset
                hourIcon = 'partly-cloudy-night'
        else:
            if InterPhour[idx, 13] > 0.125:
                hourText = 'Mostly Clear'
            else:
                hourText = 'Clear'

            if hour_array_grib[idx] < InterSday[hourlyDayIndex[idx], 17]:
                # Before sunrise
                hourIcon = 'clear-night'
            elif hour_array_grib[idx] >= InterSday[hourlyDayIndex[idx], 17] and hour_array_grib[idx] <= InterSday[
                hourlyDayIndex[idx], 18]:
                # After sunrise before sunset
                hourIcon = 'clear-day'
            elif hour_array_grib[idx] > InterSday[hourlyDayIndex[idx], 18]:
                # After sunset
                hourIcon = 'clear-night'

        # Show the wind icon if there is no precipitation and no fog.
        if InterPhour[idx, 2] < 0.02 and InterPhour[idx, 15] >= (1000 * visUnits):
            # Show the wind text before the sky text
            if InterPhour[idx, 10] > (6.7056 * windUnit):
                cIcon = 'wind'
                cText = 'Breezy and ' + cText
            elif InterPhour[idx, 10] > (10 * windUnit):
                cIcon = 'wind'
                cText = 'Windy and ' + cText
            elif InterPhour[idx, 10] > (17.8816 * windUnit):
                cIcon = 'wind'
                cText = 'Dangerously Windy and ' + cText
        else:
            # Show the precipitation text or fog text before the wind text
            if InterPhour[idx, 10] > (6.7056 * windUnit):
                cText += ' and Breezy'
            elif InterPhour[idx, 10] > (10 * windUnit):
                cText = ' and Windy'
            elif InterPhour[idx, 10] > (17.8816 * windUnit):
                cText = ' and Dangerously Windy'

[alexander0042]

Love all these changes! Your point about the units is really well taken, and it's something that bugs me with the existing setup. I originally set it up that way since it flowed nicely as I was writing it, and simplified some of the rounding logic; however, it probability should be changed so that we use SI throughout and the conversion is the last step.

Re: the summaries, expanding the current approach might work, but I'm thinking we'll have to create a new script that takes the forecast and then produces a summary in the Dark Sky format, then we can run it through their translation library. They have an appendix that describes the outputs that the library expects to receive, so if we can go from the existing numerical forecast to words in that format, then the translations are done

[cloneofghosts]

So for the summaries the plan is to re-write them in node/javascript and run it through the translation module and then send it back to the API script?

Do you want me to add Mostly Clear as a translation to the repo? I know Dark Sky it didn't exist and most other sites have something similar and we can setup all the conditions I listed in this issue 2y ago: https://github.com/Pirate-Weather/pirateweather/issues/3

I have a PR already created here https://github.com/Pirate-Weather/translations/pull/3 which I can just add it to

[alexander0042]

Mostly clear is a good idea! I don't want to add too many, since the languages will need to be updated for anything we add in, but that's doable. These are the current ones from here:

  "light-clouds": "partly cloudy",
  "medium-clouds": "mostly cloudy",
  "heavy-clouds": "overcast",

Plus clear, so it'd just be adding in that one line

[cloneofghosts]

Yeah, I can add it to my existing PR without any issues (I'll call it very-light-clouds). I was also thinking about Mist/Haze/Smoke but

1. Might be adding too many things
2. Not sure on how to calculate each one and I can't really find a consensus on how it should exactly to calculate it.

[cloneofghosts]

@alexander0042 I know this issue is a bit all over the place but some updates for some stuff:

Translations In my fork I've setup issue templates and a pull request template. The PR template I'm thinking of just having one checklist instead of two. (Not sure if this link works but you can try https://github.com/cloneofghosts/translations/compare/plugin-test...cloneofghosts:translations:master)

I think the issue templates are fine but would like to get some feedback on them to make sure. Maybe we can edit later once people start using them.

This repo Not sure what the plan is but I added a lint action, dependabot (would do renovate but I have to ask to add it) and codeql (I think this doesn't work because its not public yet). For lint I'm using the default setup which isn't as aggressive as the one used in the HA repo but can add more to it. I have another branch setup with the HA rules and it flagged a ton of stuff.

[cloneofghosts]

@alexander0042 With the Time Machine stuff fixed I'm assuming this is the next big priority on the list? Are we going to use this repo or the main PW repo for the code? For me personally it would make sense to have it separated so things are easier to track but if you prefer to use the main repo then that's fine and I'll delete this when that's been done.

[alexander0042]

Really excited to finally have this on the repo (including your apparent temperature fix, so we can probably delete that branch)! it was a bit of a production to get everything pulled together, so thank you for keeping the wheels on while I sorted it out.

I went through and marked off a few things from your (excellent) list, so the main things left are issue template, PR template, and issue config. For the issue template, my idea is to ask four main questions:

• Is this related to the production API (then go there!)
• Is this related to the HA integration (then go there!)
• Is this related to an ingest script
• Is this related to a response script

For the PR template, I like the idea of using the HA one, but I might simplify it down somewhat to remove things we don't have yet.

On the topic of PR's, I added a Contributor Agreement to make sure I don't break any rules publishing this as AGPL. If you have a minute, could you go through the flow and see if it works- I added it to the weather-vue repo since it has to be public to test it.

I'm going to get these created and added in next, and then hopefully we can push this live soon!

[cloneofghosts]

I went through and marked off a few things from your (excellent) list, so the main things left are issue template, PR template, and issue config. For the issue template, my idea is to ask four main questions:

• Is this related to the production API (then go there!)
• Is this related to the HA integration (then go there!)
• Is this related to an ingest script
• Is this related to a response script

For the Issue Templates you can look at what I have setup in the API/HA repo. You can a config.yml file to create links to the other repos below the templates.

For the issue templates you can look to see what I did in the other repos and set something up like that? Have one for bug, general and documentation? You can look at how I set it up on the other repos but change the fields for ones that better represent this repo? Like maybe one to put OS details, one for error log, one for if response/ingest, etc.

For the PR template, I like the idea of using the HA one, but I might simplify it down somewhat to remove things we don't have yet.

For this you can look to see what I setup in the translations repo and change what you need from there?

Another quick question - is this file needed anymore? If not then it should probably be removed - the only real changes in that file are fixes to make sure that some data points are in the expected range.

As for the apparent temperature fix - haven't looked at the file but you need to make sure you fix both locations where its being calculated and not just one of them.

[cloneofghosts]

On the topic of PR's, I added a Contributor Agreement to make sure I don't break any rules publishing this as AGPL. If you have a minute, could you go through the flow and see if it works- I added it to the weather-vue repo since it has to be public to test it.

I think this was still setup on the translations repo? When dependabot ran updates it commented on the PRs it created and while it didn't comment on my PR (probably bc it was created before it was added) it does show up in the list of checks that its successful.

[alexander0042]

Apparently so! I also set that one, but it didn't initially connect- I guess it just took a minute.

Did dependabot work after you went through the CLA flow? I can also add things accounts manually for things like it

[cloneofghosts]

I tried to sign that PR by going through the flow but it didn't work. It must only work for the person that submitted the PR. I can try a dummy one on either repo to see if it works?

[alexander0042]

Re: the latest push with 2.4.0 fixes- it's so much better to have version control for this, so that's a win. However, I should have done it as a branch that could be PR'd in, so I'll do that next time

[cloneofghosts]

Re: the latest push with 2.4.0 fixes- it's so much better to have version control for this, so that's a win. However, I should have done it as a branch that could be PR'd in, so I'll do that next time

I think I have branch protection rules setup for the master branch but its not enabled because the repo is private.

Has this issue been fixed at all? https://github.com/Pirate-Weather/pirateweather/issues/360

I know cloudCover can go over 100%, RH can go above 100% and visibility can below 0. There may be others which can have similar issues so might be good to check the other data points and see.

Forgot to mention also this issue: https://github.com/Pirate-Weather/pirateweather/issues/369

[cloneofghosts]

I've added PR/Issue templates to this repo if you could take a look and modify as needed. Wasn't sure what exactly to add but I think its a good enough start for now and we can modify as needed in the future.

[cloneofghosts]

@alexander0042 Just wanted to check in and see if this issue is good to close? I'm not sure if you've looked at the issue/pr templates and want to make sure they're good before closing.

Also just wanted to point out that while I normally merge in the dependency updates in the other repos on this one I assigned them to you. With this sort of stuff I know that potentially updating dependencies can break things so I don't want to accidentally break something after merging. (Plus I don't have a system capable of running the code atm)

Oops, accidentally clicked the close with comment button :laughing:

[alexander0042]

Lets keep it open for a couple more days- I want to quickly look over those templates to see if there's anything I can improve, although I doubt it! Just wanted to start with the translations stuff though, since it's really exciting, and building some new images now (with node and the additional Python packages) to test