pelias / wof-admin-lookup

Who's on First Admin Lookup for the Pelias Geocoder
https://pelias.io
MIT License
9 stars 24 forks source link

replace parallel-transform with pipeline-pipe #321

Open tadjik1 opened 2 months ago

tadjik1 commented 2 months ago

:wave: I did some awesome work for the Pelias project and would love for everyone to have a look at it and provide feedback.


Here's the reason for this change :rocket:

Hey, I've noticed that the adminLookup stream doesn't end with the end of pbf parser and found an [https://github.com/mafintosh/parallel-transform/issues/4](untucked issue in the parallel-transform module). Someone came up with the solution by creating new module pipeline-pipe with slightly different syntax, but solving the same problem. I've update dependency and tested with my local setup, I hope this would be helpful for you as well.


Here's what actually got changed :clap:

Replace parallel-transform dependency with pipeline-pipe and converted callback-based transform function inside lookupStream with promise-based.


Here's how others can test the changes :eyes:

Now, you can handle end of the stream in pelias-openstreetmap module:

// default import pipeline
streams.import = function () {
  streams
    .pbfParser()
    .pipe(streams.docConstructor())
    .pipe(streams.addressesWithoutStreet())
    .pipe(streams.tagMapper())
    .pipe(streams.addressExtractor())
    .pipe(streams.blacklistStream())
    .pipe(streams.categoryMapper(categoryDefaults))
    .pipe(streams.addendumMapper())
    .pipe(streams.popularityMapper())
    .pipe(streams.adminLookup())
    .on('end', () => console.log('ended'))
};

In my case I'm not using Elasticsearch (and pelias-dbclient) and import data into different source after extraction/mapping.

missinglink commented 2 months ago

Hi @tadjik1 thanks for the PR.

I'm trying to understand better, the issue you linked seems to suggest that the end event is emitted correctly at the end, but the finish event is emitted too early.

In your example you have .on('end', () => console.log('ended')), didn't this always work? Is the problem with finish or end?