Improve file sync API for the operators

Background

The current solution for file sync to the vehicles is not optimal and requires the operators to build their own software solution for syncing. This leads to the operators having different routines for synchronizing files and different type of errors may occur.

See current spec here: https://adt.transhub.io/3.x/packages/

Some operators also question the need to synchronize all media content, as it might not be applicable for the transportation mode they are providing. Eg. safety videos for boats are now synced to buses

Ruter also have a business requirement to make the synchronization solution more flexible, to allow for campaign creations on same day.

Suggestion

Provide an easier and more standardized way of synchronizing media files using a HTTP webserver and Rclone directly to each vehicle (or back office).

The backend providing the files will be a standard http server, which may also serve as a simple hosting solution for vehicle setups without local webservers.

The media files are to be stored in folder based on the names of the screen configs, in addition to some shared folders which are to be synchronized to all vehicles.

Benefits

No need for specialized software by each operator.
Allow each operator to synchronize only the files needed for their vehicle setup
Improve stability of synchronization from todays mixed environments
Low data transfers during sync may allow operators to sync directly from each vehicle
makes it possible to display dpi-app directly from the internet with all the associated media files, if desirable.

Example

rclone.conf

[ruter-dpi-app]
type = http
url = http://example-host.transhub.io/ruter/

Syncing of the DPI-app to www/app-folder (on-board vehicle)

rclone sync --create-empty-src-dirs dpi-vehicle-display:app www/app

Syncing of shared media / resources, and media intended for screen config 1:

rclone sync --create-empty-src-dirs dpi-vehicle-display:media/audio www/media/audio
rclone sync --create-empty-src-dirs dpi-vehicle-display:media/1 www/media/1
rclone sync --create-empty-src-dirs dpi-vehicle-display:resources www/resources

These should be setup to run as cron-expressions every X minutes.

File structure on Ruters HTTP server:

.
├── app
│   ├── 20b2975909da159ea2d3.woff2
│   ├── 2b3e1faf89f94a483539.png
│   ├── 33c5a6438e61067e958e.woff2
│   ├── 416d91365b44e4b4f477.png
│   ├── 430.4bf1b6f6a784390c6ad1.bundle.js
│   ├── 430.4bf1b6f6a784390c6ad1.bundle.js.map
│   ├── 49064197253970513594.woff2
│   ├── 67c7ca0e0fdc64af8add.svg
│   ├── 830.4bf1b6f6a784390c6ad1.bundle.js
│   ├── 830.4bf1b6f6a784390c6ad1.bundle.js.LICENSE.txt
│   ├── 830.4bf1b6f6a784390c6ad1.bundle.js.map
│   ├── 8f2c4d11474275fbc161.png
│   ├── 9a0a8712d99ed5f48a5d.woff2
│   ├── a6b9ffbd0a5d54c381b3.woff2
│   ├── app.4bf1b6f6a784390c6ad1.bundle.js
│   ├── app.4bf1b6f6a784390c6ad1.bundle.js.LICENSE.txt
│   ├── app.4bf1b6f6a784390c6ad1.bundle.js.map
│   ├── baeeab7d65eb57d6644e.woff2
│   ├── c07c7a79ff9b69084415.woff2
│   ├── changes.md
│   ├── favicon.ico
│   ├── fd6fbf6e20af7c9fd79d.woff2
│   ├── main.html
│   ├── oslo
│   │   └── map.tiles
│   ├── vendor.4bf1b6f6a784390c6ad1.bundle.js
│   ├── vendor.4bf1b6f6a784390c6ad1.bundle.js.LICENSE.txt
│   └── vendor.4bf1b6f6a784390c6ad1.bundle.js.map
├── media
│   ├── 1
│   │   ├── 1080p_Ruter_Flowers.mp4
│   │   ├── 1080p_Ruter_Holdning_Glad.mp4
│   │   ├── 1080p_Ruter_Protes.mp4
│   │   ├── 1080p_Toyen_K2.mp4
│   │   ├── 30_dager_299_16x9.mp4
│   │   ├── 30_dager_499_16x9.mp4
│   │   ├── 480p_Ruter_Holdning_Glad.mp4
│   │   ├── 480p_Toyen_K2.mp4
│   │   ├── 720p_Ruter_Holdning_Glad.mp4
│   │   ├── 720p_Toyen_K2.mp4
│   │   ├── Boreal_safety_video.mp4
│   │   ├── buss-husk-bilett.mp4
│   │   ├── bytt-1.mp4
│   │   ├── bytt-2.mp4
│   │   ├── campaign34.html
│   │   ├── Gratis_om_bord_skjorta.mp4
│   │   ├── Mikromobilitet_Scooter_Tier.mp4
│   │   ├── Mikromobilitet_Scooter_Voi.mp4
│   │   ├── Mikromobilitet_Strand_Tier.mp4
│   │   ├── Mikromobilitet_Strand_Voi.mp4
│   │   ├── pristiltak-ali.mp4
│   │   ├── pristiltak-nora.mp4
│   │   ├── Pristiltak_pris_16-9_voksen.png
│   │   ├── REIS_Klar_10s.mp4
│   │   ├── REIS_Klar_still.png
│   │   ├── ruter-app-1-mid.mp4
│   │   ├── ruter-app-2-mid.mp4
│   │   ├── ruter-app-3-mid.mp4
│   │   ├── ruter-app-4-mid.mp4
│   │   ├── ruterapp_still_1.png
│   │   ├── ruterapp_still_2.png
│   │   ├── ruterapp_still_3.png
│   │   ├── ruterapp_still_4.png
│   │   ├── RUTER_Gratist_16x9_10s_Soppla.mp4
│   │   └── ruter-sommerkampanje.mp4
│   ├── 2
│   ├── 3
│   ├── 4
│   │   ├── 1080p_Ruter_Flowers.mp4
│   │   ├── 1080p_Ruter_Holdning_Glad.mp4
│   │   ├── 1080p_Ruter_Protes.mp4
│   │   ├── 1080p_Toyen_K2.mp4
│   │   ├── 30_dager_299_16x9.mp4
│   │   ├── 30_dager_499_16x9.mp4
│   │   ├── 480p_Ruter_Holdning_Glad.mp4
│   │   ├── 480p_Toyen_K2.mp4
│   │   ├── 720p_Ruter_Holdning_Glad.mp4
│   │   ├── 720p_Toyen_K2.mp4
│   │   ├── Boreal_safety_video.mp4
│   │   ├── buss-husk-bilett.mp4
│   │   ├── bytt-1.mp4
│   │   ├── bytt-2.mp4
│   │   ├── campaign34.html
│   │   ├── Gratis_om_bord_skjorta.mp4
│   │   ├── Mikromobilitet_Scooter_Tier.mp4
│   │   ├── Mikromobilitet_Scooter_Voi.mp4
│   │   ├── Mikromobilitet_Strand_Tier.mp4
│   │   ├── Mikromobilitet_Strand_Voi.mp4
│   │   ├── pristiltak-ali.mp4
│   │   ├── pristiltak-nora.mp4
│   │   ├── Pristiltak_pris_16-9_voksen.png
│   │   ├── REIS_Klar_10s.mp4
│   │   ├── REIS_Klar_still.png
│   │   ├── ruter-app-1-mid.mp4
│   │   ├── ruter-app-2-mid.mp4
│   │   ├── ruter-app-3-mid.mp4
│   │   ├── ruter-app-4-mid.mp4
│   │   ├── ruterapp_still_1.png
│   │   ├── ruterapp_still_2.png
│   │   ├── ruterapp_still_3.png
│   │   ├── ruterapp_still_4.png
│   │   ├── RUTER_Gratist_16x9_10s_Soppla.mp4
│   │   └── ruter-sommerkampanje.mp4
│   ├── 5
│   ├── audio
│   │   ├── safety_announcement.ogg
│   │   ├── short_stopplace.mp3
│   │   └── stopplaces
│   │       ├── nsr_stopplace_5054.mp3
│   │       └── nsr_stopplace_5055.mp3
│   ├── b1
│   ├── b2
│   │   ├── 1080p_Ruter_Flowers.mp4
│   │   ├── 1080p_Ruter_Holdning_Glad.mp4
│   │   ├── 1080p_Ruter_Protes.mp4
│   │   ├── 1080p_Toyen_K2.mp4
│   │   ├── 30_dager_299_16x9.mp4
│   │   ├── 30_dager_499_16x9.mp4
│   │   ├── 480p_Ruter_Holdning_Glad.mp4
│   │   ├── 480p_Toyen_K2.mp4
│   │   ├── 720p_Ruter_Holdning_Glad.mp4
│   │   ├── 720p_Toyen_K2.mp4
│   │   ├── Boreal_safety_video.mp4
│   │   ├── buss-husk-bilett.mp4
│   │   ├── bytt-1.mp4
│   │   ├── bytt-2.mp4
│   │   ├── campaign34.html
│   │   ├── Gratis_om_bord_skjorta.mp4
│   │   ├── Mikromobilitet_Scooter_Tier.mp4
│   │   ├── Mikromobilitet_Scooter_Voi.mp4
│   │   ├── Mikromobilitet_Strand_Tier.mp4
│   │   ├── Mikromobilitet_Strand_Voi.mp4
│   │   ├── pristiltak-ali.mp4
│   │   ├── pristiltak-nora.mp4
│   │   ├── Pristiltak_pris_16-9_voksen.png
│   │   ├── REIS_Klar_10s.mp4
│   │   ├── REIS_Klar_still.png
│   │   ├── ruter-app-1-mid.mp4
│   │   ├── ruter-app-2-mid.mp4
│   │   ├── ruter-app-3-mid.mp4
│   │   ├── ruter-app-4-mid.mp4
│   │   ├── ruterapp_still_1.png
│   │   ├── ruterapp_still_2.png
│   │   ├── ruterapp_still_3.png
│   │   ├── ruterapp_still_4.png
│   │   ├── RUTER_Gratist_16x9_10s_Soppla.mp4
│   │   └── ruter-sommerkampanje.mp4
│   ├── b3
│   │   ├── bike_timelaps_9_16.mp4
│   │   └── norled-sikkerhet-oslofjorden-vertical.mp4
│   ├── manifest.js
│   ├── manifest.json
│   ├── t2-left
│   ├── t2-right
│   └── t3
└── resources
    └── stop-requested-2020-11-02T15-02-50Z.mp3

First of all - great idea. I think this makes a lot of sense.

Some feedback:

How will this change https://adt.transhub.io/3.x/packages/ ? Will it continue to be supported, but with somewhat different paths? Or is the plan to phase out the existing service? If so, when?

The mechanism where Sporveien (and others) can download only what they need for their particular vehicle/screen type is great. It will save bandwidth, storage space - and with a large number of vehicles: Ultimately money.

"Rclone" was probably selected because "Rclone" can be used to access (for example) S3 and other file storage services. This means Ruter will only need to provide credentials to PTOs - no servers or other infrastructure.

At the same time: This requires every PTO to to install a new software component ("rclone") where they want to retrieve the files. For servers, that's probably easy. For old embedded operating systems - less so. Could Ruter please consider using Rsync over SSH an an alternative?

Rsync/ssh is generally available regardless of OS (and OS age). FWIW: My opinionated view is that rsync is a more "standard" way to transfer files
Rsync transfers less data (looks at parts of a file), while Rclone only deals with change at the file level (file changed - whole file is sent again).
Some Cloud Providers have been unhappy with Rclone in the past (https://www.theregister.com/2017/05/23/amazon_drive_bans_rclone_storage_client/). Rclone has also been implicated in illicit activities (https://en.wikipedia.org/wiki/Rclone#Rclone_and_cybercrime). There might be a somewhat higher chance of some cloud provider shutting down Rclone access due to how it is used. This could pose a risk.

Feeback to other items:

No need for specialized software by each operator.
The safety regime for SL18 will always mean that updates are downloaded to a backend server. Then they are tested on simulator. Then they are deployed to vehicles in traffic. It will never be 100% automatic. That's just the safety and approval regime used - it's the "law". So the SL18 backend team will need change the current functionality used to download media/dpi and make new packages available in the SL18 "content distribution" system.
Allow each operator to synchronize only the files needed for their vehicle setup
Great!
Improve stability of synchronization from todays mixed environments
Likely. We don't really see this.
Low data transfers during sync may allow operators to sync directly from each vehicle
Safety regime. Won't be allowed. That said - it'd likely be even lower if using "rsync" instead of "rclone". :)
makes it possible to display dpi-app directly from the internet with all the associated media files, if desirable.
Might make a lot of sense for some PTOs.

I am not sure what you mean by not needing a local webserver. Are you intending to run the browser from file://? That would seem strange to me as you would have to download to all TFTs instead of one local source. Or is the intention to run the web application and sources over the mobile network? I don't think that is a good solution in general, caching on a local web server in the vehicle is better in my opinion.

How would you know which folders to bring locally as it depends on how you configured the TFTs?

We use rsync for this today instead of rclone, else that is basically what we do.

Using a zip files however makes sure you don't get file name encoding issues across platform so keep that in mind.

I second the use of rsync instead of rclone. Less dependencies, easier to use on various targets and stable.

As mentioned above we would also always sync to our backend server and then push to the vehicles, so it doesn't really make things easier, mostly work for the sake of changing.

For us, just lettings us now which files are unnecessary to sync would be use, like the structures, but since it creates a dependency on configuration the sign in the vehicle, you would likely end up syncing all not to have to maintain a match.

We have thought about lazy loading to the vehicle instead, based on the first request from the DPI application.

@audunf

How will this change https://adt.transhub.io/3.x/packages/ ? Will it continue to be supported, but with somewhat different paths? Or is the plan to phase out the existing service? If so, when?

The intention is to develop the new synchronization service in cooperation with the operators and allow our users to opt-in in to the new solution in the beginning. Meaning both solutions will be working for a period of time. The existing service will most likely be a downstream result from the new service, so that we ensure the resulting files are the same everywhere.

How long we will keep the existing service around really depends on the usage from the operators. I doubt that we can turn off the existing solution until all operators either agree to do so or have signed on to ADT 4 or 5 (I'm guessing 2025/26?).

"Rclone" was probably selected because "Rclone" can be used to access (for example) S3 and other file storage services. This means Ruter will only need to provide credentials to PTOs - no servers or other infrastructure.

One of the reasons we find Rclone attractive is that we might avoid having to deal with credentials all together. Our suggestion is to put up a public web-server which operators may synchronize against. Ref: https://rclone.org/http/

My opinionated view is that rsync is a more "standard" way to transfer files

This is 100% correct. Rsync is great and should be your go-to-tool for most data transfer operations between trusted parties. We do think however that running a public accessible rsync daemon server might pose a risk. And running a private rsync server requires us to spend time managing keys. (it is also non-trivial for us to setup a rsync daemon server in our kubernetes environment, while setting up a web-server is something we do all the time)

That being said, we have not concluded that Rsync is out of the picture, just that it requires more work and maintenance from our side.

@NiclasLindgren

I am not sure what you mean by not needing a local webserver. ... Or is the intention to run the web application and sources over the mobile network?

yes, this is the correct interpretation. This is not going to be the recommended setup for our existing fleet of vehicles any time soon.

But we keep the door open for more "light-weight" dpi installations in case there is a demand for it in the future. E.g. displays managed by other commercial entities like Clear Channel or simpler vehicle setups like in mini-vans and such.

How would you know which folders to bring locally as it depends on how you configured the TFTs?

Is this something you wished you had more information about? As an api? You probably will get this information from the future operator portal, but we could consider other options for sharing this information also.

We have a list of vehicleId's and what displays / configs they are supposed to be setup to use, so we could consider using this information in some way.

@eivindga I forget that the content is public information. :)

Right. So rclone itself won't be mandatory - just one of several possible ways to synchronize the data.

Key takeaways:

There will be a public HTTP(S) endpoint serving the files
Each PTO should poll this X times per day (realistically: Once per day)
There will be directories for app and for the different screen sizes/configurations. Each PTO pulls what they need
Files must be unpacked to the on-board/local web server in the same directory structure they are found (mention that?)

On reading this again I see that you state that the transport protocol is HTTP, and that rclone is a suggestion. Could be an idea to emphasize HTTP even more, and just mention rclone as one of several options to sync the content towards the end of the text? On first reading, rclone appeared mandatory due to the emphasis. Very specific, somewhat fringe mandatory tools are a red flag. HTTP is the opposite. :)

@audunf

There will be a public HTTP(S) endpoint serving the files

Yes

Each PTO should poll this X times per day (realistically: Once per day)

I think our ambition is having this more frequent than once a day. That way we are able to create content and have it available on vehicles within minutes, not days.

There will be directories for app and for the different screen sizes/configurations. Each PTO pulls what they need

Yes, exactly

Files must be unpacked to the on-board/local web server in the same directory structure they are found (mention that?)

Yes, I will try to clarify that in the original description.

Regarding Rclone being mandatory and fringe tool, I would argue it is pretty widely used and especially well suited for this task. 42k stars on Github and supporting a plethora of ways the operators can synchronize files.

Examples:

On board the vehicle:

HTTP -> to local files

Operator backoffice:

HTTP -> FTP / SFTP / SMB ++ to vehicles
(optionally) HTTP -> to local files & Rsync to vehicles

By configuring multiple remote endpoints using the technology of your choice, you may synchronize files between the remotes with a single command from your backend:

Rclone config:

[ruter-dpi-app]
type = http
url = http://example-host.transhub.io/ruter/

[SL18-tram01]
type = sftp
host = 172.102.0.123
user = sporveien
key_file = /home/bruxy/.ssh/id_rsa

Sync files between the two remotes:

rclone sync --create-empty-src-dirs ruter-dpi-app:app SL18-tram01:www/app

I think our ambition is having this more frequent than once a day. That way we are able to create content and have it available on vehicles within minutes, not days.

Understandable ambition, and this could absolutely be more agile. Current process is:

New DPI app/media is made available in Trikken's TEST environment
The new content is pushed to a simulator (affectionately referred to as the "TIB" - Tram In a Box), where it is validated on several test journeys using old/stored GPS data, simulated odometer, door lock/open data etc.
Once tested and approved, necessary paperwork is filled out. For changes to software and configuration files there are several gating meetings. I'm a bit hazy on the particular requirement for the DPI app - pretty sure they don't differentiate
New DPI app/media is rolled out to actual vehicles via the "Content Update" system. I think it is unlikely that this will change, or that it needs to: The system is flexible and can push content any time.

If there is a wish to speed up/remove the test, paperwork, and approval parts then you'll need to talk to "Sporveien Trikken" - likely the Tech. Reponsible and the Safety & Security group. There will need to be arguments about how Ruter "owns" this part of the experience, and not Sporveien. Might be they don't see it quite that way. :-) For vehicles that are in operation (out driving) this might be a particularly hard sell. They'll ask what is gained by taking what will be perceived as a substantial increase in risk.

Might be that it could be approved as a "standard change".

RuterNo / adt-doc