save most recent full traffic dump somewhere

[joostschouppe]

During the process, we create a huge file in memory that has all the traffic signs This data is filtered and the result is pushed to maproulette For data quality assurance and for other projects, it would be useful to also save the full dump somewhere. Just the most recent one would do (to limit storage, and to have a permanent URL of the file) Ideally, we use an FTP server for this (Github isn't great for storage) All we need on this end is to configure a Secret and add some lines of code to save the file after downloading

[jbelien]

[...] Github isn't great for storage

That's actually not true. GitHub offers 2 ways of storing (large) files:

• Storing workflow data as artifacts
• Git Large File Storage

If I understand correctly, that dump file is generated during the process using GitHub Actions, right ? That's exactly what Artifacts are made for : storing GitHub Actions results (for a maximum of 90 days).

If that limit of 90 days is not enough, we can simply store that dump file in the repository itself by enabling Git Large File Storage (LFS) (for a maximum of 2Go).

I would like to avoid to setup a FTP server on our server. Technically, it is of course not an iss1ue (we only use 50% of the storage on our server - which has a storage of 25Go) but security wise I would like to avoid to open a door on that server to avoid potential security breaches.

Questions:

• How large is that dump?
• Is 90 days retention enough for what you have in mind?
• If not, is that file smaller than 2Go (see previous question)?
• If still not, then I would consider sFTP with limitations to ensure security on our server (hoping that file is not bigger that 10Go - see previous question).

[joostschouppe]

90 days is OK, as it's the lack of officially available simple dump that is the main reason we want this. So old data is not interesting. File size should probably be below 2 gig. So it sounds like using an Artefact would be great. Example from Road Completion: https://github.com/osmbe/road-completion/actions/runs/5636286952 The only downside is that there's no permanent URL, but that's not critical

[jbelien]

Awesome! Let me know if you need me to update your GitHub Actions.

What you need to use is https://github.com/marketplace/actions/upload-a-build-artifact

[ivandiaz-tomtom]

Sure, we can use upload artifacts to GitHub. That should be straight forward.