persistent queue

[derhuerst]

A persisting queue will allow the script to fail and/or be cancelled, and resumed later. It will also make hafas-generate-gtfs work in memory-constrained environments, or with very large areas.

Changes necessary:

1. [ ] Specify all tasks (departures fetching, trips fetching) in a JSON-serialisable format. (['fetch-trip', tripId, lineName], etc).
2. [ ] Implement an in-memory store for stops, lines & trips (see API below).
3. [ ] Switch from queue to a persisting one. (I'd like to try bullmq.) Also mention in the readme that Redis is required.

store API

{
    hasStop: async id => Boolean,
    getStop: async id => Stop,
    putStop: async (id, stop) => {},
    // same with lines & trips
}

[asuender]

Hello. I saw you started a project in Node.JS which provides conversations between the HAFAS raw data and GTFS format, just like me. May I help you with your todo's? I first thought to build a script that generates GTFS data from the output of the functions provided by your package 'hafas-client'.

PS: I'm pretty new in writing Node.JS code - but fairly interested in trains.

[derhuerst]

I saw you started a project in Node.JS which provides conversations between the HAFAS raw data and GTFS format, just like me.

If you mean this project, no: This project fetches data from a HAFAS instance and generates GTFS. It does not parse the HAFAS-Rohdatenformat (which is unfortunately proprietary and not publicly documented).

I first thought to build a script that generates GTFS data from the output of the functions provided by your package 'hafas-client'.

That is what this package does. It expects the API client to have the same API as hafas-client@5 (or be hafas-client@5).

May I help you with your todo's?

We can collaborate! I will create a tracking Issue with a to-do list.

[derhuerst]

I've updated the original post with a small to-do list. You're welcome to contribute!

[asuender]

Specify all tasks (departures fetching, trips fetching) in a JSON-serialisable format. (['fetch-trip', tripId, lineName], etc).

Assuming to that, we only store relevant data in a JSON-like format and leave unnecessary things out. Is that what you wanted to say? When we fetch departures, trips etc. we receive much more data than needed.

[derhuerst]

Specify all tasks (departures fetching, trips fetching) in a JSON-serialisable format. (['fetch-trip', tripId, lineName], etc).

Assuming to that, we only store relevant data in a JSON-like format and leave unnecessary things out. Is that what you wanted to say?

Yes.

[asuender]

I will go through your code a bit for understanding -this may take some time.

Specify all tasks (departures fetching, trips fetching) in a JSON-serialisable format. (['fetch-trip', tripId, lineName], etc).

I'm going to try solving this part.