metafacture / metafacture-core

Core package of the Metafacture tool suite for metadata processing.
https://metafacture.org
Apache License 2.0
69 stars 34 forks source link

Enable wormholes to work with `flowtail` #482

Open TobiasNx opened 1 year ago

TobiasNx commented 1 year ago

At the moment I am not able to combine a -tee command with a wormhole since the wormhole only works with flow but not with flowtails the "inside" of a tee.

Would be nice enable flowtail to work with wormhole. It is not urgend.

"https://raw.githubusercontent.com/langsci/opendata/master/catalog.csv"
| open-http(accept="application/csv")
| as-lines
| decode-csv(hasHeader="true", separator="\t")
| stream-tee
| {
  fix(FLUX_DIR + "langSciPress_csv_data.fix", *)
    |stream-to-triples
    |@X
  }{
    fix(FLUX_DIR + "langSciPress_csv_id.fix", *)
    | literal-to-object
    | open-http
    | as-records
    | match(pattern="(name=\"DC)\\.(\\w*?)\\.(\\w*?)(\")", replacement="$1_$2_$3$4")
    | match(pattern="(name=\"DC)\\.(\\w*?)(\")", replacement="$1_$2$3")
    | read-string
    | decode-html
    | fix(FLUX_DIR + "langSciPress.fix", *)
    | stream-to-triples
    | @X
  };

@X
| wait-for-inputs("2")
| sort-triples(redirect="true")
| collect-triples
| encode-json
| validate-json(output_schema, writeValid=metadata_valid, writeInvalid=metadata_invalid)
| oersi.OersiWriter(backend_api, user=backend_user, pass=backend_pass, log=metadata_responses)
;