search

cbs228 / sameold

SAME/EAS decoding for Rust 🦀
Apache License 2.0
25 stars 4 forks source link

sameold: location API #1

Open cbs228 opened 3 years ago

cbs228 commented 3 years ago

At present, sameold only supports retrieving the six-digit SAME location codes (PSSCCC). No further interpretation is performed. We should consider:

Download locations for GIS files, as of this writing. These files are subject to periodic update and should not be used blindly.

A-c0rN commented 2 years ago

Hello, if you're interested I have some JSON files I am going to attach with some SAME Location Codes listed. I hope this helps! Canada.json.txt United_States.json.txt

cbs228 commented 2 years ago

Hello @A-c0rN, can you tell me a little bit more about your planned use-case? Are you just looking to display placenames, or did you have something more complicated in mind? Polygons are useful but would greatly increase the size of the dataset.

I think it would be better to source placenames from official government websites and use scripts to convert them to the format we need. This will help ensure the process is repeatable and consistent. xastir uses this approach (see script). If digesting shapefiles is required, ogr2ogr is very useful. Bash would probably be easiest, but an argument could be made for python.

Once we have them in an intermediate format, we'll commit them here for use in builds.

I think it would be a good idea to bake the placenames into the executable with phf. The codes change seldom enough that producing new binaries each time that happens isn't a big deal… and it's much more convenient than needing a config file. Do you think the placenames should be their own crate?

We'll need to make sure to include special marine area codes and "partial county" codes.

cbs228 commented 2 years ago

@A-c0rN, I'm still looking for some details on your planned use-case to make sure my changes will cover it. Does this proposal meet your needs?

A-c0rN commented 2 years ago

I didn't have a planned Use Case, I was just submitting the biggest list of all known FIPS codes we know of.

Also, I would love to be able to generate polygons, but as of the way SAME works, it's impractical. For example, how would one define "105007" or "Northwest Benton County"? What would that look like?

Those files I provided were to allow to be modified as a location database for easy FIPS to County/Province name checking. For example, You could have a flag that SameDec can call in a child that has "FIPS County Names", to allow 005007 to become Benton County.

Providing a location database which maps codes to place names. In the United States, there are many codes for marine areas and other places which are not part of core FIPS, and there are several different lists.

This is what I was doing.

cbs228 commented 2 years ago

For example, how would one define "105007" or "Northwest Benton County"? What would that look like?

The NWS publishes shapefiles for partial-county alerts. This file has a few entries for partial counties, including 132003 for "Northwest Clark" county in Nevada. There is only one entry for Benton county. It is not subdivided in this dataset.

The NWS has a fact sheet for partial-county alerting. It suggests that only a handful of forecast offices have it implemented. I am not a broadcast engineer… but I believe that the SAME spec merely provides suggestions for how partial counties MAY be split. It is not an authoritative specification for where county parts are or what they should be called. That spec is probably the above shapefile.

Someone please correct me on this if I am wrong.

Have you (or anyone else) observed partial-county alert codes that are not defined in the above shapefiles?

In my area, I have observed production TORs that still alert entire counties. The actual alerts disseminated by my forecast office are storm-based warnings: the warned area is an arbitrary polygon. The polygon may only cover part of a county, but the SAME message will still target the whole county. This is probably because there is simply no agreed-upon SAME code for partial counties here.

Including polygons would complicate things. The maximum size of a crate on crates.io is 10 MB, but the shapefile weighs in at 30+ MB. I wouldn't be able to distribute the polygon database as part of my source code. There would have to be some other distribution mechanism, and I'd have to ingest it at runtime.

Placenames are a lot less data and would be easier to accomplish.

A-c0rN commented 2 years ago

The NWS publishes shapefiles for partial-county alerts. This file has a few entries for partial counties, including 132003 for "Northwest Clark" county in Nevada. There is only one entry for Benton county. It is not subdivided in this dataset.

It appears, from talking to some NWS employees, That they're trying to make a push for more offices to use subdivisions, but cooperation from a per-office area isn't there yet. This dataset probably will expand over time, hopefully.

In my area, I have observed production TORs that still alert entire counties. The actual alerts disseminated by my forecast office are storm-based warnings: the warned area is an arbitrary polygon. The polygon may only cover part of a county, but the SAME message will still target the whole county. This is probably because there is simply no agreed-upon SAME code for partial counties here.

The SAME spec allows per-county subdivisions in the first number in the FIPS code "PSSCCC" as P = Subdivision, SS = State Code, CCC = County Code. The SAME Spec defines the codes, but the actual AREAS are still being defined, Such as Benton County, AR still not having defined subdivisions.

Have you (or anyone else) observed partial-county alert codes that are not defined in the above shapefiles?

Funnily enough, The shapefile is a really good way to show which NWS offices use subdivisions or not. The reason you don't see alerts on the non-subdivided counties for subdivisions is that the NWS office that serves that area doesn't use them. From what I have heard, however, This may change as the NWS looks into making more offices use subdivisions.

Placenames are a lot less data and would be easier to accomplish.

Those JSON files I uploaded are every FIPS code (Without subdivisions) and their corresponding placename, including US, Canada, US Outlying Islands, and Marine. These files allow a lookup table, such as if you receive a SAME code "ZCZC-EAS-DMO-005007+0100-0050348-WACN -", You could remove the FIPS code "005007", and then check the last 5 digits "05007" to those files to get "Benton, AR". You can also use the first digit "0" to check the subdivision,

In your original post, you said:

In the United States, there are many codes for marine areas and other places which are not part of core FIPS, and there are several different lists.

What I have done in these files is precompiled every list of FIPS codes I could get my hands on, and then made "Master Lists" for the U.S. and Canada containing every FIPS to PlaceName code I could. It is the "Lists of Lists", you could say.

They also include "00000 = All of the U.S. / All of Canada" and 99999 = Test Location

Canada.json.txt > All Canadian FIPS codes to PlaceNames United_States.json.txt > All United States FIPS codes to PlaceNames

cbs228 commented 2 years ago

Good to know. As mentioned previously, I am looking to build a script which fetches this data from authoritative sources / official government websites. The script will build the list in a way that is repeatable. It would help me more if you could assemble a list of URLs where this data may be found. The exact formatting is unimportant.

I think for now I'm not going to worry about polygons, but shapefiles are still okay as data sources.

How did you want this to work in samedec? I presume you want it to set an environment variable on started processes:

SAMEDEC_LOCATION_NAMES="Arkansas:Benton County;Oklahoma:Delaware County;…"

The information will also be available via a Rust API.

I have no plans at present to add dsame-like fulltext generation.

A-c0rN commented 1 year ago

An example is ID AMZ158 → SAME 075158 The last three digits nnn map directly to a SAME code suffix.

This is true for codes that go by "xxCyyy", but not xxZyyy. The NWS has two different objects in their database, "Zones" and "Counties". Counties map to FIPS directly, but Zones do not. Example: Milwaukee County in Wisconsin maps to WIZ066, but is FIPS and County code 079. (055079 / WIC079)

The NWS do have a list of Zones to County mappings, I believe.

cbs228 commented 1 year ago

AFAIK, the zones are only for coastal marine areas. These "zones" can cover very large areas of water. Regular forecast zones don't map to SAME.

By the way, v0.3.0 is out and makes substantial changes to the way bursts are assembled into messages.

RCGV1 commented 2 months ago

For now I am using a csv file full of USA location codes to decode which county an alert is for.

cbs228 commented 2 months ago

For now I am using a csv file full of USA location codes to decode which county an alert is for.

I have been using the most excellent dsame python module to translate SAME messages into "human-readable" text. This works well enough for me that I'm not really pursuing a built-in location database or text generator at this time. Maybe someday.