kalisio / krawler

A minimalist (geospatial) ETL
https://kalisio.github.io/krawler/
MIT License
54 stars 13 forks source link

Transform options erased in readCSV hook #248

Closed claustres closed 1 year ago

claustres commented 1 year ago

Steps to reproduce

Use the readCSVhook with the following options:

{                                                
  header: true,                                  
  delimiter: ';',                                
  transform: {                                   
    omit: [ 'opening_date', 'pack' ],            
    unitMapping: { station_id: [Object] }        
  }                                              
}

Expected behavior

JSON transformation is applied.

Actual behavior

JSON transformation is not applied.

In the hook the options seem to have been modified by Papa.parse() when retrieved in the end event of the stream:

{                                                
  header: true,                                  
  delimiter: ';',                                
  transform: false,                              
  dynamicTyping: false                           
}

We should probably clone options before sending it to Papa.parse().

System configuration

Tell us about the applicable parts of your setup.

Module versions (especially the part that's not working): 2.4

NodeJS version: 16

Operating System: Windows