Investigate inactive stops in stops.csv

[james-westwood]

Having re-looked at the stops.csv data recently, I noticed there is a column called Status which indicates if the stop is in use or not.

The possible entries are

• "" (i.e. it's blank)
• "active"
• "inactive"
• "new"
• "pending"

In our current stops_df (22nd Feb, 2022, running from the master branch) does not contain this column, nor has it been filtered on this basis.

So far it has proven difficult to find a definition of Status values.

• NaPTAN guide for data managers does not contain a defnition of the Status field.
• The document "STOP STATUS IN NaPTAN AND DEALING WITH NaPTANs THAT ARE NO LONGER REQUIRED" contains definitions of "active", "suspended" etc, but says they should be under "Stop Availability". Whereas the "Status" columns should have the values "ACT", "DEL" and "PEN". Also note that this document is marked as DRAFT.

Overall the task is to remove stops that are not in use from the analysis. This probably means filtering out "inactive" and "pending" but for thoroughness we need to get definitions on the others.

Steps for the developer:

• email the DfT NaPTAN team to enquire about:
  • If we are using the most recent data (our current link), and if not, should we be using the NAPTAN API for example? There is a new Beta Site should we be downloading from that?
  • If we should use the API (could be better for us) can we have some documentation on it?
  • What the definitions of "active", "inactive" etc are.
• Look at either our existing data (from the csv) or the new data they point you towards
• Filter the dataframe to remove stops that are not in use.

Bear in mind our stops data must have the longitude data and the latitude data.

[Antonio-John]

Hi,

I work within the Office for National Statistics and am using NaPTAN data to conduct some analysis. I had a couple of queries about the data which I was hoping you could answer:

1) The data we currently use is from the NaPTAN website. Is this the most recent data or should we be using the NaPTAN api?

2) If we should be using the API, is there any user guide or documentation that would be helpful for this?

3) There is a column called Status within the stops data. I have looked at the NaPTAN guide for data managers and couldn’t find a definition of the Status field. The unique entries in the Status field appear to be:

Possible Values “”(i.e. it's blank) "active" "inactive" "new" "pending"

Do you have a definition of what the values mean please? This will help us remove stops that are not in use from our analysis. 

Many thanks,

Antonio`

[Antonio-John]

draft email written. Would you mind taking a look to see if it's okay please? @james-westwood @Mark-Simons-ONS Then can send after that

[james-westwood]

Except the formatting it all looks good. You could say who the data analysis is for (the SDG team).

[Antonio-John]

great, have sent email :)

[Antonio-John]

Summary of Email:

• The website and API both have the same output and they are always most up to date.
• Limited documentation on API page.
• Newest user guide on data saved here.
• The only three variables that should be in the status field is "Active", "Inactive" & "Pending"
• Pending means stop is missing, or flagged as deleted from the most recent data upload, and may be in process of being made inactive. Will continue to be exported as if ‘active’ until status is clarified.

Couple of thoughts I had from this email.

• Are we using the latest version of stops? We might have to pull from the API to ensure we are? Not sure if this is a yearly publication or how often the data get's updated. Might have to time stamp data to ensure analysis is reproducible.
• Can definitely get rid of inactive stops.
• Not sure about pending. I'm not sure what the NaPTAN team means by will continue to be exported as if active.

[Antonio-John]

Link(https://naptan.app.dft.gov.uk/Reports/frmStopsSummaryReport) which shows last time each LA was updated.

[jwestw]

I suggest going back and asking for those couple of points of clarification. 1) How often is the data published? -> I see this might be answered by your last comment. 2) How can we tell when the data was updated (is there a time stamp or publication date?) > I see this might be answered by your last comment. 3) Could they explain what "exported as if active mean"? Can transport users catch transport from Pending stops or not?

[Antonio-John]

-Yesterday morning I went back and asked about how frequently the data is published. The answer is the timeframes of the data being published isn't consistent. Each LA uploads to the data when and if they need to. The link above shows when a LA last uploaded or changed something in the data. They say a lot of users upload every week but there is the potential that nothing has changed. -I will go back and ask about exported if active.

[Antonio-John]

Conversation between @Antonio-John & @james-westwood 25/02/2022

• Start to use the NaPTAN API to be able to download data. New ticket for this #183
• Can drop inactive stops from stops.csv, waiting for confirmation how to deal with pending stops.

[Antonio-John]

-Treat stops without a status (i.e blank) as active. -Treat stops that are pending as active as well. Only 11 stops like this in the live dataset as of the 1st March 15:00.

[jwestw]

This has been implemented by #186