zskarte / zskarte-client

Zivilschutz-Karte allows to draw situation maps for disaster management
https://www.zskarte.ch/
MIT License
13 stars 7 forks source link

feat(csv layer): add option for layers generated from csv files #426

Closed swerder closed 1 month ago

swerder commented 6 months ago

Add an option to directly load csv data with coordinates as layer with point features. Also allowing filtering the the data from csv before internally create the features.

As this change is based on the geojson logic and settings you can also activate search for the generated features here.

This function was designed to have a possibility for an offline search for places, and use the "swissNAMES3D_PLY.csv" from https://www.swisstopo.admin.ch/en/landscape-model-swissnames3d (with filtering for "OBJEKTKLASSE_TLM" = "TLM_SIEDLUNGSNAME")

It is based on geojson update https://github.com/zskarte/zskarte-client/pull/417

some words to CSV parsing

Most used CSV parser (also @fast-csv/parse, was already in source) are for node runtime and would need pollyfill for node-stream to work in web. Therefore I add uDSV. It's not as much used, but is small and work as expected on my tests. If you planned to add stream pollyfill anyway. The parse logic in GeoJSONService.fetchCsvData could be changed to:

import { parse, Row } from '@fast-csv/parse';

        const csvLines: Row[] = [];
        const stream = parse({ headers: true, ignoreEmpty: true })
          .transform((data) => ({
            ...data,
            [feature.fieldX]: parseFloat(data[feature.fieldX]),
            [feature.fieldY]: parseFloat(data[feature.fieldY]),
          }))
          .on('error', (error) => console.error(error))
          .on('data', (row) => {
            if (isNaN(row[feature.fieldX]) || isNaN(row[feature.fieldY]) || (row[feature.fieldX] === 0 && row[feature.fieldY] === 0)) {
              return;
            }

            if (regexPatterns?.length) {
              const matches = regexPatterns
                .map((pattern) => row[pattern.field].match(pattern.regex))
                .filter((match) => match) as RegExpMatchArray[];
              if (!matches || matches.length !== regexPatterns.length) {
                return;
              }
            }
            csvLines.push(row);
          })
          .on('end', (rowCount: number) => console.log(`Parsed ${rowCount} rows`));
        stream.write(csvContent);
        stream.end();
swerder commented 1 month ago

@thegnuu here you merged the swerder branch (my dev branch to verify all my changes merged together) and therefore also add the changes from the commit "apply config to auto build swerder branch" I don't have problem with that but I'm not sure if it make sense to have them in your code base.

thegnuu commented 1 month ago

@swerder This happened more by accident than by intention to be honest, I am still not sure why pulling the upstream dev into this PR branch pulled your other changes as well, but I am pretty sure the issue was o my end somehow ;) Anyways I double checked most of the stuff and realised that it might have been easier than merging every change separately.

I am not finished with cleaning things up since I did some dependency-updates as well and I will need to test everything once the server is ready.

Thank you for letting me know anyways, I will remove the unused stuff once I am done.

Unfortunately I started updating the server to strapi 5 until I realised that one of our dependencies is not maintained anymore, so I guess we will need to find a solution for this one at the next "WK". Maybe I will just copy the plugin into our project and update it appropriately, but I need to check out what I have to do first, since I never use strapi.