Produce a "things to check out" in user logs?

[jteresco]

Thinking about some of the emails I get from users when they send in .list updates, where they find the errors in their lists from highway data updates over the course of a few days, it might be nice to have a report for users indicating which routes they have in their lists that have recent updates entries.

[michihdeu]

https://github.com/TravelMapping/Web/issues/204 ?

[jteresco]

Yes, thanks, I forgot we had that Issue over there.

Thinking about it just a bit, it seems the user log entries are easier to get in place, since the information about what list entries a user has are no longer available when we get to the Web interface.

Let's leave both Issues open for now. It's not something I'll be looking at for a few months in any case.

[yakra]

• [x] Create a root string -> Route object hash table
• Move Reading updates file to before Processing traveler list files
• As we go, populate a set of Routes that are recently updated. I imagine this would be done using the datetime module; I'd have to look up the specifics.
• While processing list files, track a set of recently updated routes for each traveler.
• When we match a Route object to a Traveler's list entry, see if it's in the recently_updated set. If so, check whether it's in the t_recently updated set.
  • If not, add it to the set, and append "NOTE: [Route.readable_name] has been recently updated" (perhaps with the date of its most recent update) to TravelerList.log_entries. These would appear amid the error reports for .list lines, E.G. "Unknown region/highway combo in line", "Waypoint label(s) not found in line", "Incorrect format line", etc.
  • If so, skip over the note, and suppress duplicate reports for the same Route.

[michihdeu]

We do currently have more than 3,000 update entries. I think it's worth to think about improving the process medium-term.

I observe list file updates a little bit... Well, just because I wanna see whether people add my new systems drafted... Nevertheless, I see that people only update their list file when entries are broken which is indicated in user log file but rarely because of update entries... Minimum for European updates...

It falsifies stats......

[jteresco]

Other quick thoughts

Probably not hard: Include only information about updates entries from the most recent week/month/6 months?

Probably too hard and maybe not reasonably possible: Supress entries for updates entries for which .the list file already has fixes.

[yakra]

We do currently have more than 3,000 update entries. I think it's worth to think about improving the process medium-term.

Yes, something like https://github.com/TravelMapping/Web/issues/203 would be useful IMO.

[michihdeu]

Probably not hard: Include only information about updates entries from the most recent week/month/6 months?

Just a very small improvement. It's almost the same like today when you read top-down.

Probably too hard and maybe not reasonably possible: Supress entries for updates entries for which .the list file already has fixes.

How should that be detected?

We could add an filter by region / system as we already have. Or an automated filter showing update entries from regions contained in the list file of the selected user only. Not route but region-specific because we also add new routes!

[michihdeu]

Yes, something like TravelMapping/Web#203 would be useful IMO.

Ah, I second what that clever guy has suggested two years ago 🤣

[michihdeu]

I thought that it might make things easier when we would have an user login on the site which might also be necessary once we would have automated site updates after submitting list files.... We could send private notifications to the user account....

Not sure... maybe a bad idea...

[yakra]

Probably not hard: Include only information about updates entries from the most recent week/month/6 months?

Just a very small improvement. It's almost the same like today when you read top-down.

If Jim meant including that info in a filtered table on the Web end, yes, agreed. It's only a little more difficult right now to pick 10-20-40-80-160 entries, whatever I need, until I have entries threu the desired date. But an improvement nonetheless. If Jim meant including updates information for user log entries, that's what I was getting at in my 3rd bullet point above.

Probably too hard and maybe not reasonably possible: Supress entries for updates entries for which .the list file already has fixes.

How should that be detected?

With the "plain English" updates text, I don't see any way of correlating updates entries to .wpt lines, waypoint labels, or .list lines, short of implementing a new updates.csv line format. With >3000 entries already, that looks like a non-starter. I agree with "Probably too hard and maybe not reasonably possible".

[yakra]

Another possibility: At the end of a log file, list all traveled routes that have known entries in updates.csv, most recent first.

Traveled 37 of 294 (12.6%), Clinched 0 of 294 (0.0%) active systems
Traveled 2 of 100 (2.0%), Clinched 0 of 100 (0.0%) preview systems

NY NY17M last updated 2020-10-28
NY NY272 last updated 2020-10-28
CZE I16 last updated 2020-09-21
IRL R472  last updated 2020-05-02
NH NH121A last updated 2020-02-10
NY NY163 last updated 2019-03-11
KS KS7 last updated 2018-11-21
ME ME188 last updated 2017-05-18

etc.

[michihdeu]

Another possibility:

I think you mean all routes where the user has travels - not all routes in general?

• How could I "clean" the list? Mark it checked? FP list in user list file - sounds crazy but it would be doable 😄
• What's about new routes or even systems in a region I have travels?
• Removed routes are already notified in the first lines of the log file. The text could be changed from "unknown..." to "removed route".

[yakra]

I think you mean all routes where the user has travels - not all routes in general?

Yes, intended to say that. Edited my post.

How could I "clean" the list? Mark it checked? FP list in user list file - sounds crazy but it would be doable 😄

No cleaning. It'll be sorted by date; users can just remember when they last looked at it. :) (Some people leave comments in their .list files.)

What's about new routes or even systems in a region I have travels?

Any new routes, a user would have to travel them for it to matter.

Removed routes are already notified in the first lines of the log file. The text could be changed from "unknown..." to "removed route".

There's no way to map what routes listed in updates.csv as removed to whatever may be in a .list file: https://github.com/TravelMapping/HighwayData/blob/a91132e90520162c9ddc278ad9f170c45f9785cb/updates.csv#L3551 2020-11-17;(USA) Kansas;Turner Diagonal;;Deleted route mentions neither KS TurDia nor ks.turdia. There's no longer any info in the database (or rather, the in-memory structures in siteupdate that eventually become the database) about what KS TurDia in a .list file means or used to mean. The error Unknown region/highway combo in line error message will be sufficient.

[jteresco]

I'm happy to see us thinking about this again. I think it would be a huge usability improvement to include those entries in the user logs. Once seen there, the user can then search appropriately in the updates for details. We can also think about ways to help track those down.

I agree that deleted routes are sufficiently represented by the current error message. It would be nice to augment them somehow if there's an updates entry related, but as @yakra points out, we just don't have that information in the data.

[yakra]

Quote from: @jteresco on Feb 6, 2019:

Include only information about updates entries from the most recent week/month/6 months?

The average .list file is updated every 34.5 days. How many travelers update at least every... week	month	6 months
38	117	232
13.1%	40.3%	80%

Travelers who update or look at their log files less often than our cutoff wouldn't benefit from this, thus inspiring my thoughts above, something more permanent that travelers can refer back to. With this new way of thinking about doing things, is it still possible to do the inline notifications for recently updated routes within some threshold? Yes. (Easier in fact that what I brainstormed above; that post needs a rethink, mumble mumble...) Should we still do that additionally?

• Pro: Right there at the top of the log file along with errors, sure to grab the user's attention.
• Con: Some users might not like being unable to control whether or not any junk appears above Processed ### good lines marking ##### segments traveled. in their .log file.

---

If going back to the beginning of TM's history to list all routes' updates is too long and going back to some arbitrary cut-off is not long enough, there's the option of listing what's changed since the most recent .list update. Beginning, end, both, neither, of a .log file, whatever's clever...

• This info can be gathered by localupdate.sh before launching siteupdate.py.

[yakra]

Rethinking an earlier post...

• [x] For sanity/organizational purposes, address #353 first.
• ~Have an adequate understanding of Python's datetime module or whatever we need... ;)~
• [x] During Route construction, set last_update to None.
• [x] Move Reading updates file to before Processing traveler list files.
  • [x] If a root is provided in an entry, look it up in root_hash.
  • [x] If we get a result, set that Route's last_update as needed.
• [x] Include the date ~(and time?)~ of last .list update at beginning of log if available
• [x] While processing .list lines, track a set of each user's routes_traveled (init during TravelerList ctor).
• [x] Do this within Route::store_traveled_segments to eliminate redundancy.
  • [x] Python: ready to do now
  • [x] C++: Meh, pass ofstream log by reference?
• [x] When a Route object is first added to this set, see if it's within our "recent" threshold.
  • If so, append "NOTE: Route.readable_name updated on last_update" to TravelerList.log_entries. These would appear amid the error reports for .list lines, E.G. "Unknown region/highway combo in line", "Waypoint label(s) not found in line", "Incorrect format line", etc.
• And then at the end of the log file...
• [x] Remove Routes where last_update is still None from the set.
• [x] Sort Routes by last update.
• [x] Write last update for each route, with full description this time.
  • ~Suppress any "new route", "added route" or "route added" entries, case insensitive?~ No!
• [x] Clear the user's routes_traveled set.
• [x] C++: Reorganize functions/ directory

Easy to implement without having to learn new stuff. :)

[michihdeu]

I'd prefer having the beginning of the file "clean" so that I can see the important things - especially when I run data check to make sure that I have a correct list file (after editing this) and to compare the new stats with the previous stats from the site. But that's only me.

I see two options to keep the list of "things to check out" short:

1. Limit the list to e.g. 30 days and add the link to https://travelmapping.net/devel/updates.php "for older entries"
2. Introduce a key word to the user list file which indicates the last update check and only output newer entries, e.g. Last update check: 2020-11-22 (without # of course) Don't output this line as error. If the key word is found, the list is just as long as the time span is - shorter or longer. If the key word is not found, go with option 1

[yakra]

I'd prefer having the beginning of the file "clean" so that I can see the important things - especially when I run data check to make sure that I have a correct list file

Yes, this is important. Let's keep this clean and easy to see. This means...

• [x] At a minimum, leave out the update descriptions. No problem, let's do it!
• [ ] At most, leave out everything.

  Log file created at: 2020-11-21 21:52:22.378118
  Waypoint label I-40 not found in line: NC US17 SC/NC I-40
  Processed 545 good lines marking 9569 segments traveled.

• [ ] Middle ground? Keep minimal cleanly formatted info, like maybe

  Log file created at: 2020-11-21 21:52:22.378118
  Route updated 2020-10-28: NY NY17M
  Route updated 2020-10-28: NY NY272
  Route updated 2020-09-21: CZE I16
  Route updated 2020-05-02: IRL R472
  Route updated 2020-02-10: NH NH121A
  Route updated 2019-03-11: NY NY163
  Waypoint label I-40 not found in line: NC US17 SC/NC I-40
  Route updated 2018-11-21: KS KS7
  Route updated 2017-05-18: ME ME188
  Processed 545 good lines marking 9569 segments traveled.

  All uniformly formatted and lined right up in columns, with lots of whitespace so normal error messages stand out, to draw the eye right down to the Processed ... good lines line... If anything is included at the beginning, then yes, definitely limit the list.

and to compare the new stats with the previous stats from the site. But that's only me.

Keeping things DIFFable is worthwhile, sure. Although, highway data updates will naturally cause some variation, e.g. changes to overall mileage in a region, changes to one's own mileage as traveled routes are relocated, etc.

I see two options to keep the list of "things to check out" short:

Keeping the list short at the end of the log may not be important though. This is a log file, with its reams of text logging dozens of routes in dozens of systems in dozens of regions. michih.log is 13056 lines; if we filter out the zero-mileage systems it'll still be 12827 lines. Filtering & drilling down to specific routes and ranges of dates (or systems, or regions..) interactively is better left to the Web side of things. In the user log files, I'm fine with singing the Spam song with the Vikings. :)

2. Introduce a key word to the user list file which indicates the last update check and only output newer entries, e.g. Last update check: 2020-11-22 (without # of course) Don't output this line as error. If the key word is found, the list is just as long as the time span is - shorter or longer. If the key word is not found, go with option 1

Dislike. Last update check: 2020-11-22 is 4 fields, just like a standard RG Rte Wp1 Wp2 style line. I rewrote the .list line parsing routines in spring and am not keen to do so again. :) Sure, valid travel lines require 4 or 6 fields; maybe 2 fields (LastUpdate 2020-11-22) or 3 fields (Set LastUpdate 2020-11-22) could configure a thing, but what if we want to mark MA MA117 or MA MA117 I-495? If <4-field lines were introduced, they should be part of a well thought out, cohesive and extensible scheme; these lines should be reserved for future use.

1. Limit the list to e.g. 30 days

30 days (or whatever other timeframe) could be too long for some users and too short for others. Middle ground I'm thinking of is, limit it to whenever the .list file was last updated. Have localupdate.sh (and datacheck.sh?) create a file listing the most recent update time for each .list file, and pass it along to siteupdate.py

and add the link to https://travelmapping.net/devel/updates.php "for older entries"

Nah. A .log file is its own self-contained plain text thing. We can leave the web links for those who are actively navigating the web anyway. People should have no trouble finding the updates link on the TM page header.

[yakra]

@michih wrote:

How could I "clean" the list? Mark it checked?

I wrote:

30 days (or whatever other timeframe) could be too long for some users and too short for others. Middle ground I'm thinking of is, limit it to whenever the .list file was last updated.

This would in effect make the list "self-cleaning". Update the .list file, and the items there before would be considered "checked" so to speak, and thrown out. Any .list update will do, even just a comment like #Last updated 2020-11-23 before lunch.

---

My current thinking...

• Short list at beginning of log file, inline with error messages.
  • Route updated 2020-02-10: NH NH121A format, no description. Short, uniformly formatted in columns, easy on the eyes.
  • Limited to route updates since the last .list update. Using michih.log as an example, 581 routes have been updated, ever. At over 600 commits, that's on average < 1 route update notices in michih.log at any given time.
• Long list at end of log file
  • 2019-09-02 | ns.ns245 | NS NS245 | Extended eastward from NS 227 to NS 4. format.
  • Useful for users who may have missed some route updates between .list submissions, or who want to refer back to historical updates, including those from before the feature's introduction.
  • The average .list file would have 115 route update lines at the end, out of a total of 3100 lines, 3.7%
• For both of these, only routes explicitly .listed would be shown, not anything credited through concurrency detection. If I've clinched ME US202 but not .listed any ME ME4, I'd see me.us202's updates and not me.me004's.

[michihdeu]

Update the .list file, and the items there before would be considered "checked" so to speak, and thrown out.

Does an "update" without any change work? I guess a "dummy modification" is required e.g. editing a comment

[yakra]

Does an "update" without any change work?

Short answer: No, AFAIK. Empty commits are possible, but they don't make changes to any files, so no commits will be showing up in the git log for any specific file. (Unless there's something super arcane that I'm missing...)

I guess a "dummy modification" is required e.g. editing a comment

Seems to be the case, yes.

[michihdeu]

ok, go for it! 👍

[yakra]

Woo-hoo!

[yakra]

• Phase 1: long list at end of file. This is coded and (mostly) ready to go.
• Phase 2: short list at beginning of file. A few more details to be sorted out, so this can wait a bit.

Originally, I though I would sort the long list newest-first. But then it hit me... Instead of finding where this list begins within pages of text, won't people just hit Ctrl+End (or whatever does the trick in their browser or text editor) to get to the end of the file? If we instead sort the updates newest-last, the info will be right there at the end of the file without having to hit Page Up... however many times. Any objections?

[michihdeu]

Sorting: newest-last is fine me for the long list at the end of the file. When switching to short list at the beginning of the file, newest-first will be better.

However, I disagree with the two phases! Why do you wanna bother "normal" users twice? Once they are familar with phase 1, another change. No....

[yakra]

The idea is to have both the short list and the long list. I mean yeah, I didn't say "and" or "or" in this comment but I thought the idea was apparent by then...

So there will be no switching to the short list; it's a matter of adding it in. These entries will be sorted in the same order the routes are encountered in the .list file, same as the error messages and notes it's mixed in with.

As for the Phase 1 / Phase 2 distinction, it doesn't have to be anything more than how I organize the workflow & commits on my own machine.

[yakra]

short list

sorting out the how-tos; ping @jteresco

• [x] We'll need to create a file storing the most recent .list update dates. This is easy enough.

  for u in *; do echo -e `git log --pretty=%cs -n 1 $u`"\t$u"; done > $filename

• [x] ~This example uses tabs as a delimiter. We could instead use ; and treat this more like a standard TM CSV.~ Using spaces.
• ~If so, include a header row?~
• [x] We could hardcode the filename ~(as with regions.csv) or allow it to be specified via command line (as with systems.csv)~.
• ~If the latter, what commandline switch to use?~
  • ~-L, listupdatesfile~
  • ~-r, recentlistfile~
  • ~-m, mostrecentfile~
  • ~something else~
• [x] If file is unspecified, not found, or somehow does not contain a valid date for a given .list file, don't output a short list.
• [x] After siteupdate.py completes, do we have localupdate.sh...
  • ~Store it somewhere in /home/tmp/tm?~
  • [x] Or just delete it? Now that a UserData commit # is included in siteupdate.log, we can always get this info from git.

[yakra]

for u in `ls ~/TravelMapping/UserData/list_files/ | cut -f1 -d.`; do
  update=`git log -n 1 --pretty=%cs $u.list`
  echo -en "$u\t$update\t" | tr - '\t'
  echo -en `grep -m 1 'good lines' ~/TravelMapping/yakra/DataProcessing/siteupdate/cplusplus/_upd/logs/users/$u.log | cut -f2 -d' '`"\t"
  update=`echo $update | tr -d -`
  tail -n +$(expr $(grep -m 1 -n '^Most recent updates for listed routes:' ~/TravelMapping/yakra/DataProcessing/siteupdate/cplusplus/_upd/logs/users/$u.log | cut -f1 -d:\
                   ) + 1\
            ) ~/TravelMapping/yakra/DataProcessing/siteupdate/cplusplus/_upd/logs/users/$u.log \
  | cut -f1 -d'|' | tr -d - | sed "s~.*~(&>= $update) \* $update~" | bc | grep -v '^0$' | wc -l
done

[jteresco]

* [ ]  If so, [include a header row](https://github.com/TravelMapping/DataProcessing/issues/88#issuecomment-449204162)?

Probably. I think most CSV files would have that as part of their format.

* [ ]  We could hardcode the filename (as with regions.csv) or allow it to be specified via command line.

* [ ]  If the latter, what commandline switch to use?

  * [ ]  -L, listupdatesfile
  * [ ]  -r, recentlistfile
  * [ ]  -m, mostrecentfile
  * [ ]  something else

* [ ]  If file is unspecified, not found, or somehow does not contain a valid date for a given .list file, don't output a short list.

* [ ]  After siteupdate.py completes, do we have localupdate.sh...

  * [ ]  Store it somewhere in /home/tmp/tm?
  * [ ]  Or just delete it? Now that a UserData commit # is included in siteupdate.log, we can always get this info from git.

I think a fixed, temporary filename is fine, and I don't think there's need to keep it around.

[jteresco]

I admit to being a little behind on the thinking about this. So is the idea to include all/only seemingly relevant highway data updates since the last time the .list was updated? In many cases, that's probably pretty reasonable, but I know speaking for myself, I don't look at the .log file on each .list update. Especially while traveling, I like to make my new entries daily but I tend not to look much at the .log for things other than the entries I just made until I'm done.

That said, any functionality that helps people find relevant updates more easily is a huge step forward.

[michihdeu]

@jteresco I fully agree. That's why I suggested here

Introduce a key word to the user list file which indicates the last update check and only output newer entries, e.g. Last update check: 2020-11-22 (without # of course) Don't output this line as error. If the key word is found, the list is just as long as the time span is

Dislike. Last update check: 2020-11-22 is 4 fields, just like a standard RG Rte Wp1 Wp2 style line. I rewrote the .list line parsing routines in spring and am not keen to do so again. :) Sure, valid travel lines require 4 or 6 fields; maybe 2 fields (LastUpdate 2020-11-22) or 3 fields (Set LastUpdate 2020-11-22) could configure a thing, but what if we want to mark MA MA117 or MA MA117 I-495? If <4-field lines were introduced, they should be part of a well thought out, cohesive and extensible scheme; these lines should be reserved for future use.

I never ever suggested a fixed key word though. Just any (optional) key word so that I can manually indicate the date when my last full check happened.

[jteresco]

Anything in the logs is potentially helpful. I don't worry much about clutter. Log files are going to be big and messy.

The functionality that would probably help users most is https://github.com/TravelMapping/Web/issues/204 so they could filter however they want.

Let's not overcomplicate the initial version of log file entries, whether it's 30 days or since the last .list update, it will be helpful to users.

[yakra]

@jteresco wrote:

I admit to being a little behind on the thinking about this. So is the idea to include all/only seemingly relevant highway data updates since the last time the .list was updated?

The idea is to try for a "best of both worlds" approach, and have the info available in 2 places.

• The "short list" at the beginning of the .log, mixed in with error messages & notices, as the relevant .list lines are encountered. Limited to route updates since the last .list file update. No update text, only date & route name. Lots of whitespace, so users can easily scan down to Processed [0-9]+ good lines marking [0-9]+ segments traveled.

In many cases, that's probably pretty reasonable, but I know speaking for myself, I don't look at the .log file on each .list update. Especially while traveling, I like to make my new entries daily but I tend not to look much at the .log for things other than the entries I just made until I'm done.

• This is where the "long list" at the end of the file comes in, after the grand summary. You won't need to worry about missing any info between successive .list updates; the most recent update entry will be listed in full for every route traveled, back to the beginning of time. They will be sorted earliest to most recent, so just hit Ctrl+End or whatever and you'll see your most recently route update(s) right at the end of your .log file.

In both cases "seemingly relevant highway data" means the most recent updates entry for explicitly .listed routes only. If I mark ME US202 as clinched but don't mark ME4 at all, I'll see updates entries for US202 but not ME4. (If an update to ME4 were to be relevant to my travels, it would have to be on the part concurrent with US202, right?)

That said, any functionality that helps people find relevant updates more easily is a huge step forward.

I truly believe this will be a best of both worlds approach. Quick info at our fingertips in the part of our .log files we most frequently check for errors or notices, with noise kept to a minimum. Not enough to be disruptive for people who update semi-frequently. Once a traveler becomes aware something has been updated and may need attention, they can hit Ctrl+End and check out the most recent updates in more detail at the end of the file. Or head on over to the web as appropriate.

---

@michih wrote:

Introduce a key word to the user list file which indicates the last update check

I never ever suggested a fixed key word though.

Uhmm... huh? If you want to manually indicate when your last check happened, you're of course free to use a regular comment to do that.

---

@jteresco wrote:

Anything in the logs is potentially helpful. I don't worry much about clutter. Log files are going to be big and messy.

Exactly. They're already thousands of lines. Thus I believe the long list at the end is no problem. Log files are by their nature all the everything.

The functionality that would probably help users most is TravelMapping/Web#204 so they could filter however they want.

Agreed. The web is where we drill down & filter. I just spent a few minutes trying out SQL queries , so I'm thinking about implementing it. :) (And don't forget TravelMapping/Web#203.)

---

Back to the main point of this post for a moment...

I'd prefer having the beginning of the file "clean" so that I can see the important things - especially when I run data check to make sure that I have a correct list file

If we reeeally want to, It'd be easy as pie to completely leave out the short list altogether from logs produced by datacheck.sh, and only have localupdate.sh produce the file we'll need. But there's not even much to filter out at this point; It could be that the majority of "data checkers" would want to keep the info there. Except for @rickmastfan67, those of us with accounts on noreaster to use datacheck.sh update our lists frequently enough to have no more than 1 or 2 short list entries. I hope that 1 or 2 short list entries where .log errors would be are few and unobtrusive enough to cope with.

user	short list size	comments
@Duke87ofST uke87	1
mikeandkristie	0
@rmihailucian	0
@jteresco	1
@mapcat	0
@markkos1992	0
@yakra	0
@neroute2	1
@si404	1
@rickmastfan67	23	.list last updated Tue Jul 16 01:04:16 2019 -0400
@michihdeu	1
@ovoss	2

[yakra]

A few different factors led to using spaces as delimiters in listupdates.txt. So I'm naming it that, rather than as a .csv. Since it's a .txt file that only temporarily exists while running localupdate.sh, I'm not bothering with a header line.

Here's what the beginning of michih.log looks like on lab2:

Log file created at: Wed Dec  2 15:48:44 2020
michih.list last updated: 2020-12-01 22:03:35 -0500
Route updated 2020-12-01: CZE D6
Route updated 2020-12-01: CZE I6
Route updated 2020-12-01: CZE I16
Route updated 2020-12-01: CZE II606
Processed 5131 good lines marking 60862 segments traveled.
Clinched Highway Statistics

I can't put these online because I've been swearing at 2 different routers for 2 hours now. Can't have another go at it till much later this evening.

[yakra]

2018-01-16 | nh.nh016abar | NH NH16ABar | Deleted. This route can still be tracked as NH 16A (Bartlett). Don't pull the readable_name from the Route object. Instead, use the field from updates.csv In this case, seeing US 302 Business (Bartlett) in context makes more sense.

---

harvey.log: 2019-01-06 | wy.neentrd | WY NEEntRd | Route deleted (replaced with Northeast Entrance Road) vs updates.csv: 2019-01-06;(USA) Wyoming;US 212 (Yellowstone National Park);wy.neentrd;Route deleted (replaced with Northeast Entrance Road)

[rickmastfan67]

The reason I really haven't updated my list file is because I haven't really gained any mileage to update it with. The most mileage I've gained in the last 4 years was during the Pittsburgh road meet in '19.

Could you link me the log entry for my file and I'll just take a quick look at it to see if I need to adjust anything.

[yakra]

Log file created at: Thu Dec  3 23:22:02 2020
rickmastfan67.list last updated: 2019-07-16 01:04:16 -0400
Note: deprecated route name I-285FutWin -> canonical name I-285 in line: NC I-285FutWin 87(I-85) NC8Lex
Note: deprecated route name I-285FutWin -> canonical name I-285 in line: NC I-285FutWin NC8_S 193A(I-40)
Waypoint label US52_N not found in line: NC I-85BLLex I-85_S US52_N
Route updated 2020-04-27: FL US17
Route updated 2019-09-05: FL US92
Route updated 2020-05-23: FL US98
Route updated 2020-05-19: FL US441
Route updated 2020-03-21: GA US41
Route updated 2020-03-21: GA US341
Route updated 2020-05-23: NC US29
Route updated 2020-05-23: NC US74
Route updated 2020-02-14: PA US62
Route updated 2020-10-09: PA US322
Unknown region/highway combo in line: FL US17TrkOrl I-4(88) I-4(90)
Unknown region/highway combo in line: FL US92TrkOrl I-4(88) I-4(90)
Route updated 2019-12-15: ON ON407
Route updated 2019-08-24: NC NC24
Route updated 2019-08-24: NC NC27
Route updated 2020-11-13: NC NC150
Route updated 2020-07-06: PA PA18
Route updated 2019-10-22: PA PA31
Route updated 2019-08-27: PA PA44
Route updated 2019-09-11: PA PA56
Route updated 2020-10-09: PA PA144TrkPot
Route updated 2019-08-27: PA PA150
Route updated 2020-03-04: PA PA228
Route updated 2020-08-10: PA PA381
Route updated 2020-09-28: PA PA915
Processed 467 good lines marking 5780 segments traveled.
...
2019-08-24 | (USA) North Carolina | NC 24 | nc.nc024 | rerouted onto newly constructed bypass of Troy with points between NC109_N and NC24Bus/27Bus being new location.  Former NC 24 segment is NC 24 Business
2019-08-24 | (USA) North Carolina | NC 27 | nc.nc027 | rerouted onto newly constructed bypass of Troy with points between NC109_N and NC24Bus/27Bus being new location.  Former NC 27 segment is NC 27 Business
2019-08-27 | (USA) Pennsylvania | PA 44 | pa.pa044 | Relocated onto new alignment around PA 244 (done on 10-16-2018) between *OldPA44_A and *OldPA44_B.
2019-08-27 | (USA) Pennsylvania | PA 150 | pa.pa150 | Realigned between Church St (ChuSt_W) and Hanna St (HanSt) to reflect one-way routing at PA 120 in Lock Haven.
2019-09-05 | (USA) Florida | US 92 | fl.us092 | Removed from Seminole Boulevard and French Avenue in Sanford, and relocated onto FL 46 and CR 15 between the intersections of 1st Street/French Avenue (formerly FL46_W, now 1stAve) and I-4/Seminole Blvd (SemBlvd).
2019-09-11 | (USA) Pennsylvania | PA 56 | pa.pa056 | Realigned between *OldPA56_A and *OldPA56_B around Hiner Rd (HinRd) intersection slightly southeast of PA 259.  Realigned between *OldPA56_C and *OldPA56_D north of US 22.
2019-10-22 | (USA) Pennsylvania | PA 31 | pa.pa031 | Realigned at connector road to I-70/I-76 at the Donegal Interchange (between *OldPA31_A and *OldPA31_B).
2019-12-15 | (Canada) Ontario | ON 407 | on.on407 | Extended at eastern end from Exit 135 (ON 418) to the new interchange with ON 35/ON 115.
2020-02-14 | (USA) Pennsylvania | US 62 | pa.us062 | Realigned at Allegheny River Bridge and Hunter Station Rd (HunStaRd) between *OldUS62_A and *OldUS62_B.
2020-03-04 | (USA) Pennsylvania | PA 228 | pa.pa228 | Realigned between *OldPA228 and Westminster Rd (WesRd) at new roundabout at Saxonburg Blvd (SaxBlvd_N).
2020-03-21 | (USA) Georgia | US 41 | ga.us041 | Moved waypoint US341 (now GA7_N) from downtown Perry to new US341 point (North Perry Parkway)
2020-03-21 | (USA) Georgia | US 341 | ga.us341 | Removed from Main St and Sam Nunn Blvd and relocated to North Perry Parkway around Perry.
2020-04-27 | (USA) Florida | US 17 | fl.us017 | Removed from Main Street in Zolfo Springs, and relocated onto a new 4-lane bypass to the east between the points of *OldUS17_S and *OldUS17_N.
2020-05-19 | (USA) Florida | US 441 | fl.us441 | Removed from North Marion Avenue in Lake City, and relocated onto US 90, US 41, & FL 100A, between the intersections of US 90/North Marion Avenue & CR 100A/North Marion Avenue.
2020-05-23 | (USA) Florida | US 98 | fl.us098 | Relocated 'US319_MedN' point to the northeast by 0.2 miles due to a reroute of US 319. Old location of the US 319 intersection is now 'OldCraHwy_S'.
2020-05-23 | (USA) North Carolina | US 29 | nc.us029 | NC7_W label was at wrong point
2020-05-23 | (USA) North Carolina | US 74 | nc.us074 | NC7_W label was at wrong point
2020-07-06 | (USA) Pennsylvania | PA 18 | pa.pa018 | Realigned between *OldPA18 and I-376(39). (done in 2018).
2020-08-10 | (USA) Pennsylvania | PA 381 | pa.pa381 | Relocated SugRd waypoint (Old location replaced by *OldSugRd).
2020-09-28 | (USA) Pennsylvania | PA 915 | pa.pa915 | Realigned between Palmer St (PalSt) and PA 26.
2020-10-09 | (USA) Pennsylvania | PA 144 Truck (Potters Mills) | pa.pa144trkpot | Rerouted to use new Potters Mills Gap Interchange between RedMillRd and PA144_S.
2020-10-09 | (USA) Pennsylvania | US 322 | pa.us322 | Realigned between RedMillRd and *OldUS322_B onto new four lane freeway at Potters Mills Gap.
2020-11-13 | (USA) North Carolina | NC 150 | nc.nc150 | rerouted around Kernersville using US 421 and new roadway, points between US421(222) and MainSt_W are the new routing

[yakra]

Non-ascii text displays a bit funky. This appears to be happening at the Apache level?

https://travelmapping.net/logs/users/duke87.log

2020-10-27 | isl.th060 | ISL TH60 | Relocated into new Dýrafjarðargöng (Dyra Fjords Tunnel) east of Þingeyri
2020-10-27 | isl.th622 | ISL TH622 | Extended east along former section of TH60 bypassed by Dýrafjarðargöng (Dyra Fjords Tunnel)
2020-10-27 | isl.th626 | ISL TH626 | New Route replacing former section of TH60 bypassed by Dýrafjarðargöng (Dyra Fjords Tunnel)
2020-10-28 | isl.th402 | ISL TH402 | New route replacing former section of TH417 between TH42 and Leiðarendur Cave which is still open.
2020-10-28 | isl.th417 | ISL TH417 | Removed from a now partially closed road heading west towards TH42 and relocated onto former TH407 heading south to Bláfjöll Ski Area

lab2

2020-10-27 | Iceland | TH60 | isl.th060 | Relocated into new Dýrafjarðargöng (Dyra Fjords Tunnel) east of Þingeyri
2020-10-27 | Iceland | TH622 | isl.th622 | Extended east along former section of TH60 bypassed by Dýrafjarðargöng (Dyra Fjords Tunnel)
2020-10-27 | Iceland | TH626 | isl.th626 | New Route replacing former section of TH60 bypassed by Dýrafjarðargöng (Dyra Fjords Tunnel)
2020-10-28 | Iceland | TH402 | isl.th402 | New route replacing former section of TH417 between TH42 and Leiðarendur Cave which is still open.
2020-10-28 | Iceland | TH417 | isl.th417 | Removed from a now partially closed road heading west towards TH42 and relocated onto former TH407 heading south to Bláfjöll Ski Area

A-20 (Montréal): 251.17 of 343.07 mi (73.21%)
 (QC A-20 only)
A-25: 10.36 of 31.15 mi (33.26%)
 (QC A-25 only)
A-30 (Montréal): 90.49 of 90.49 mi (100.00%)
 (QC A-30 only)
A-30 (Bécancour): 10.46 of 10.46 mi (100.00%)
 (QC A-30Bec only)

• [x] This will take some looking at canqca_con.csv with a hex editor. The broken és have 2 extra bytes.

[rickmastfan67]

Thanks @yakra. I'll pick that apart over the weekend when I have time and fix what I can/need to.

[yakra]

I didn't copy everything from the end of the file; there are route updates from before the last .list update too https://travelmapping.net/logs/users/rickmastfan67.log

...and maybe older updates for these same routes too. Hm. Edit (clarify): Older updates for these same routes may exist, but are not shown in the log. We'd have to look them up on updates.php.

• [ ] Do we want to include all updates for .listed routes in the "long list", rather than just the most recent?

[rickmastfan67]

Oh, I thought what you linked above was just from the test server, and not on the live one yet. Should have just told me it was live on the site. :P

[yakra]

The "short list" stuff that I pasted first (before the edit) is from my own server, and not live yet.

And the "long list" stuff too. It's 1:36am lol

[rickmastfan67]

Well, anyways, I've saved the log file and will pick it apart this weekend to clear up some of the issues. ;)

[yakra]

• ~Do we add short list generation to datacheck.sh? ping @michihdeu (michih.log)~

• [ ] It occurs to me that in the strictest sense, this doesn't address the issue in the OP.

  Thinking about some of the emails I get from users when they send in .list updates, where they find the errors in their lists from highway data updates over the course of a few days

  Anything that generates an error message doesn't flag a Route as traveled, and we won't see an update notification. So users won't be helped when they're wondering what's going on when they see new error messages. In cases where the Region + Route combo is valid and lookup of labels fails, we still have a valid Route object with a pointer to an updates entry. It would be helpful to include info in these cases too. 6-field entries will be ~slightly~ more complicated ~but still easy enough~.

• What to do about non-ascii text?

  • @jteresco, is there anything in BSD+Apache config that can & should be changed?
  • Should we consider allowing standard only ascii/Latin text in updates.csv?
  • [x] @yakra, check out canqca_con.csv in a hex editor

• [ ] Thinking about expanding the long list to include all updates entries for each traveled route, rather than only the most recent. At a quick estimate, the long list may get about 23% longer.

  • A user might see only a line for 2019-11-17 | (USA) North Carolina | US 17 | nc.us017 | rerouted onto newly constructed freeway connecting Pollocksville and New Bern bypasses with points between NC58_N and 136 being new location. Former US 17 segment is US 17 Business and adjust their .list file & think they're grand, without realizing there was another update on 2019-03-22 for NC US17 that also needs attention.
  • Sometimes there's even >1 update for a route in one day (FL US17, TJK E123...)

[michihdeu]

Do we add short list generation to datacheck.sh?

How long does it take? If it is less than 10 or even 20 seconds, go for it!

Should we consider allowing standard only ascii/Latin text in updates.csv?

No!

[yakra]

How long does it take? If it is less than 10 or even 20 seconds, go for it!

24 s in /home/duke87/UserData/list_files 24 s in /home/mapcat/UserData/list_files/ 27 s in /home/markkos1992/UserData/list_files/ 7 s in /home/michih/UserData/list_files/ 15 s in /home/neroute2/UserData/list_files/ 27 s in /home/oscar/UserData/list_files/ 10 s in /home/panda80/UserData/list_files 23 s in /home/si404/UserData/list_files/ 43 s in /home/terescoj/travelmapping/UserData/list_files/ 26 s in /home/yakra/TravelMapping/UserData/list_files/

Weird that there's that much variation.

[michihdeu]

That's a lot.... Can it be enabled by a parameter? sh datacheck.sh -u or something like that?

[yakra]

Just edited my post. How does 7s work for you? Yes, it can be enabled by a parameter. But, how much does the group as a whole care about this? FWIW, datacheck.sh will still produce the long list at the end of the log file.

[michihdeu]

Weird that there's that much variation.

❓ ❔ ❓ 😮

Just edited my post. How does 7s work for you?

Considering the variation...

But yeah, fine to me.

[yakra]

Meh, it's a lot more for everybody else though. Datacheck.sh already disables several features of siteupdate.py (nmp_merged, subgraphs, colocation stats, database) in order to keep time down. If anyone really wants that info, it's available at the end of the log file. The more I think about it, the more I'd like to leave it out of datacheck.sh for time & simplicity.