Open-Earth-Foundation / CityCatalyst

Open Source carbon accounting for cities
GNU Affero General Public License v3.0
20 stars 5 forks source link

feat:EF for I.7.1 and I.8.1 #652

Closed mfonsecaOEF closed 1 month ago

mfonsecaOEF commented 1 month ago

This PR contains:

lemilonkh commented 1 month ago

Some of the CSV data seems to be invalid/ not match the original structure, please check. You can see this error on the CI results of this PR:

 == 20231114094254-emissions-factors: migrating =======
Loading emissions factor folder EFDB_2006_IPCC_guidelines...
Done loading files
Finished adding publishers
Finished adding data sources

ERROR: Error
    at Query.run (/home/runner/work/CityCatalyst/CityCatalyst/app/node_modules/sequelize/lib/dialects/postgres/query.js:50:25)
    at /home/runner/work/CityCatalyst/CityCatalyst/app/node_modules/sequelize/lib/sequelize.js:315:28
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async bulkUpsert (/home/runner/work/CityCatalyst/CityCatalyst/app/seeders/util/util.cjs:16:18)
    at async /home/runner/work/CityCatalyst/CityCatalyst/app/seeders/20231114094254-emissions-factors.cjs:90:9
    at async /home/runner/work/CityCatalyst/CityCatalyst/app/node_modules/sequelize/lib/sequelize.js:507:18
    at async Object.up (/home/runner/work/CityCatalyst/CityCatalyst/app/seeders/20231114094254-emissions-factors.cjs:48:5)
EXTRA MESSAGE: invalid input syntax for type uuid: "undefined"

So it seems you are missing an UUID somewhere, or we need to adjust the parsing code again. When making incremental updates like these, please try to stay compatible to the original parsing code (seeder script) to minimize the work required to get this PR merged.

lemilonkh commented 1 month ago

Lgtm, but check for compatibility before merging if possible with @amanda-eames

mfonsecaOEF commented 1 month ago

@lemilonkh @amanda-eames Fixed the null values ​​issue

amanda-eames commented 1 month ago

the output file for EmissionsFactor.csv need to have these headings gas,region,units,reference,gpc_reference_number,actor_id,emissions_per_activity,methodology_name,metadata,year,id so need to change emisions_factor_id --> id and gpc_ref_no --> gpc_reference_number