search

hyperledger-labs / blockchain-carbon-accounting

This project implements blockchain applications for climate action and accounting, including emissions calculations, carbon trading, and validation of climate claims. It is part of the Linux Foundation's Hyperledger Climate Action and Accounting SIG.
https://wiki.hyperledger.org/display/CASIG/Carbon+Accounting+and+Certification+WG
Apache License 2.0
195 stars 104 forks source link

loading of oil and gas data #605

Closed sichen1234 closed 2 years ago

sichen1234 commented 2 years ago

@brioux I tried loading the oil and gas data. Is this correct?

sichen$ sh download.sh 
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  146k  100  146k    0     0   198k      0 --:--:-- --:--:-- --:--:--  199k
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  156k  100  156k    0     0   574k      0 --:--:-- --:--:-- --:--:--  573k
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  148k  100  148k    0     0   597k      0 --:--:-- --:--:-- --:--:--  599k
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  112k  100  112k    0     0   364k      0 --:--:-- --:--:-- --:--:--  365k
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  150k  100  150k    0     0   544k      0 --:--:-- --:--:-- --:--:--  543k
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 23.0M  100 23.0M    0     0   9.8M      0  0:00:02  0:00:02 --:--:--  9.9M
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  188k  100  188k    0     0   514k      0 --:--:-- --:--:-- --:--:--  515k
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 1857k  100 1857k    0     0  4887k      0 --:--:-- --:--:-- --:--:-- 4876k
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 13617  100 13617    0     0  65817      0 --:--:-- --:--:-- --:--:-- 66101
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  506k  100  506k    0     0   814k      0 --:--:-- --:--:-- --:--:--  816k
sichen$ ls files/
2022-og-benchmarking-report-data.xlsx           NG_PROD_SUM_A_EPG0_VGQ_MMCF_M.xls           flaring_monitor_company_stats_reported.csv
NG_PROD_SUM_A_EPG0_FGW_MMCF_M.xls           NG_PROD_SUM_A_EPG0_VGV_MMCF_M.xls           flaring_monitor_company_stats_satellite_modeled.csv
NG_PROD_SUM_A_EPG0_VG9_MMCF_M.xls           PET_CRD_CRPDN_ADC_MBBL_M.xls                flaring_monitor_detailed_observations.csv
NG_PROD_SUM_A_EPG0_VGM_MMCF_M.xls           flaring_monitor_basin_stats.csv

Then after npm install

MacBunniePro:data sichen$ sh loadOGdata.sh 

> @blockchain-carbon-accounting/data-postgres@1.0.0 dataLoader
> ts-node src/dataLoader.ts "load_product_data" "./oil_and_gas/files/2022-og-benchmarking-report-data.xlsx" "Company_basin" "--format" "Benchmark" "--source" "https://www.sustainability.com/globalassets/sustainability.com/thinking/pdfs/2022/2022-og-benchmarking-report-data.xlsx" "--skip_rows" "7"

=== Starting load_product_data ...
Parsing worksheet |████████████████████████████████████████| 100% || 24591/24591
Loading into PostgresDB |████████████████████████████████████████| 100% | ETA: 0s | 18759/18759 records
=== Loaded ./oil_and_gas/files/2022-og-benchmarking-report-data.xlsx | Company_basin: 17496 rows loaded, 1263 ignored.
 - ignored 1263 rows because: Undefined col
=== Done, we now have 17496 product entries in the DB

> @blockchain-carbon-accounting/data-postgres@1.0.0 dataLoader
> ts-node src/dataLoader.ts "load_product_data" "./oil_and_gas/files/VIIRS_Global_flaring_d.7_slope_0.029353_2017_web_v1.xlsx" "flares_upstream" "--format" "VIIRS" "--year" "2017" "--source" "https://eogdata.mines.edu/global_flare_data/VIIRS_Global_flaring_d.7_slope_0.029353_2017_web_v1.xlsx"

=== Starting load_product_data ...
dataLoader.ts load_product_data <file> [sheet]

load data from xls, xlsx, or csv file

Positionals:
  file   XLS file to load from                                        [required]
  sheet  Name of the worksheet to load from                  [default: "Data 1"]

Options:
      --help        Show help                                          [boolean]
      --version     Show version number                                [boolean]
      --dbname      The postgres database to use.
                              [string] [default: "blockchain-carbon-accounting"]
      --dbuser      The postgres user to use.                           [string]
      --dbpassword  The postgres password to use.                       [string]
      --dbhost      The postgres host to use.                           [string]
      --dbport      The postgres port to use.                           [number]
      --dbverbose   Set this flag to show the DB debug output.         [boolean]
  -v, --verbose     Run with verbose logging                           [boolean]
      --format      Data format to load                                 [string]
      --year        Year of data to load                                [string]
      --source      Source of product data                              [string]
      --name        Product name                                        [string]
      --type        Product type                                        [string]
      --unit        unit of data being loaded                           [string]
      --skip_rows   Number of rows to skip for header                   [string]
      --cellDates   Identify cell dates                                [boolean]
      --raw         Use raw spreadsheet data                           [boolean]

Error: ENOENT: no such file or directory, open './oil_and_gas/files/VIIRS_Global_flaring_d.7_slope_0.029353_2017_web_v1.xlsx'
    at Object.openSync (node:fs:585:3)
    at Object.readFileSync (node:fs:453:35)
    at read_binary (/Users/sichen/hyperledger/blockchain-carbon-accounting/node_modules/xlsx/xlsx.js:3153:44)
    at readSync (/Users/sichen/hyperledger/blockchain-carbon-accounting/node_modules/xlsx/xlsx.js:23698:69)
    at readFileSync (/Users/sichen/hyperledger/blockchain-carbon-accounting/node_modules/xlsx/xlsx.js:23738:9)
    at parseWorksheet (/Users/sichen/hyperledger/blockchain-carbon-accounting/lib/data-common/spreadsheetImport.ts:91:28)
    at importFlareData (/Users/sichen/hyperledger/blockchain-carbon-accounting/lib/oil-and-gas-data/import.ts:92:32)
    at Object.handler (/Users/sichen/hyperledger/blockchain-carbon-accounting/data/src/dataLoader.ts:190:28)
    at processTicksAndRejections (node:internal/process/task_queues:96:5) {
  errno: -2,
  syscall: 'open',
  code: 'ENOENT',
  path: './oil_and_gas/files/VIIRS_Global_flaring_d.7_slope_0.029353_2017_web_v1.xlsx'
}
npm ERR! Lifecycle script `dataLoader` failed with error: 
npm ERR! Error: command failed 
npm ERR!   in workspace: @blockchain-carbon-accounting/data-postgres@1.0.0 
npm ERR!   at location: /Users/sichen/hyperledger/blockchain-carbon-accounting/data 

> @blockchain-carbon-accounting/data-postgres@1.0.0 dataLoader
> ts-node src/dataLoader.ts "load_product_data" "./oil_and_gas/files/VIIRS_Global_flaring_d.7_slope_0.029353_2017_web_v1.xlsx" "flares_downstream_oil" "--format" "VIIRS" "--year" "2017" "--source" "https://eogdata.mines.edu/global_flare_data/VIIRS_Global_flaring_d.7_slope_0.029353_2017_web_v1.xlsx"

=== Starting load_product_data ...
dataLoader.ts load_product_data <file> [sheet]

load data from xls, xlsx, or csv file

Positionals:
  file   XLS file to load from                                        [required]
  sheet  Name of the worksheet to load from                  [default: "Data 1"]

Options:
      --help        Show help                                          [boolean]
      --version     Show version number                                [boolean]
      --dbname      The postgres database to use.
                              [string] [default: "blockchain-carbon-accounting"]
      --dbuser      The postgres user to use.                           [string]
      --dbpassword  The postgres password to use.                       [string]
      --dbhost      The postgres host to use.                           [string]
      --dbport      The postgres port to use.                           [number]
      --dbverbose   Set this flag to show the DB debug output.         [boolean]
  -v, --verbose     Run with verbose logging                           [boolean]
      --format      Data format to load                                 [string]
      --year        Year of data to load                                [string]
      --source      Source of product data                              [string]
      --name        Product name                                        [string]
      --type        Product type                                        [string]
      --unit        unit of data being loaded                           [string]
      --skip_rows   Number of rows to skip for header                   [string]
      --cellDates   Identify cell dates                                [boolean]
      --raw         Use raw spreadsheet data                           [boolean]

Error: ENOENT: no such file or directory, open './oil_and_gas/files/VIIRS_Global_flaring_d.7_slope_0.029353_2017_web_v1.xlsx'
    at Object.openSync (node:fs:585:3)
    at Object.readFileSync (node:fs:453:35)
    at read_binary (/Users/sichen/hyperledger/blockchain-carbon-accounting/node_modules/xlsx/xlsx.js:3153:44)
    at readSync (/Users/sichen/hyperledger/blockchain-carbon-accounting/node_modules/xlsx/xlsx.js:23698:69)
    at readFileSync (/Users/sichen/hyperledger/blockchain-carbon-accounting/node_modules/xlsx/xlsx.js:23738:9)
    at parseWorksheet (/Users/sichen/hyperledger/blockchain-carbon-accounting/lib/data-common/spreadsheetImport.ts:91:28)
    at importFlareData (/Users/sichen/hyperledger/blockchain-carbon-accounting/lib/oil-and-gas-data/import.ts:92:32)
    at Object.handler (/Users/sichen/hyperledger/blockchain-carbon-accounting/data/src/dataLoader.ts:190:28)
    at processTicksAndRejections (node:internal/process/task_queues:96:5) {
  errno: -2,
  syscall: 'open',
  code: 'ENOENT',
  path: './oil_and_gas/files/VIIRS_Global_flaring_d.7_slope_0.029353_2017_web_v1.xlsx'
}
npm ERR! Lifecycle script `dataLoader` failed with error: 
npm ERR! Error: command failed 
npm ERR!   in workspace: @blockchain-carbon-accounting/data-postgres@1.0.0 
npm ERR!   at location: /Users/sichen/hyperledger/blockchain-carbon-accounting/data 

> @blockchain-carbon-accounting/data-postgres@1.0.0 dataLoader
> ts-node src/dataLoader.ts "load_product_data" "./oil_and_gas/files/VIIRS_Global_flaring_d.7_slope_0.029353_2017_web_v1.xlsx" "flares_downstream_gas" "--format" "VIIRS" "--year" "2017" "--source" "https://eogdata.mines.edu/global_flare_data/VIIRS_Global_flaring_d.7_slope_0.029353_2017_web_v1.xlsx"

=== Starting load_product_data ...
dataLoader.ts load_product_data <file> [sheet]

load data from xls, xlsx, or csv file

Positionals:
  file   XLS file to load from                                        [required]
  sheet  Name of the worksheet to load from                  [default: "Data 1"]

Options:
      --help        Show help                                          [boolean]
      --version     Show version number                                [boolean]
      --dbname      The postgres database to use.
                              [string] [default: "blockchain-carbon-accounting"]
      --dbuser      The postgres user to use.                           [string]
      --dbpassword  The postgres password to use.                       [string]
      --dbhost      The postgres host to use.                           [string]
      --dbport      The postgres port to use.                           [number]
      --dbverbose   Set this flag to show the DB debug output.         [boolean]
  -v, --verbose     Run with verbose logging                           [boolean]
      --format      Data format to load                                 [string]
      --year        Year of data to load                                [string]
      --source      Source of product data                              [string]
      --name        Product name                                        [string]
      --type        Product type                                        [string]
      --unit        unit of data being loaded                           [string]
      --skip_rows   Number of rows to skip for header                   [string]
      --cellDates   Identify cell dates                                [boolean]
      --raw         Use raw spreadsheet data                           [boolean]

Error: ENOENT: no such file or directory, open './oil_and_gas/files/VIIRS_Global_flaring_d.7_slope_0.029353_2017_web_v1.xlsx'
    at Object.openSync (node:fs:585:3)
    at Object.readFileSync (node:fs:453:35)
    at read_binary (/Users/sichen/hyperledger/blockchain-carbon-accounting/node_modules/xlsx/xlsx.js:3153:44)
    at readSync (/Users/sichen/hyperledger/blockchain-carbon-accounting/node_modules/xlsx/xlsx.js:23698:69)
    at readFileSync (/Users/sichen/hyperledger/blockchain-carbon-accounting/node_modules/xlsx/xlsx.js:23738:9)
    at parseWorksheet (/Users/sichen/hyperledger/blockchain-carbon-accounting/lib/data-common/spreadsheetImport.ts:91:28)
    at importFlareData (/Users/sichen/hyperledger/blockchain-carbon-accounting/lib/oil-and-gas-data/import.ts:92:32)
    at Object.handler (/Users/sichen/hyperledger/blockchain-carbon-accounting/data/src/dataLoader.ts:190:28)
    at processTicksAndRejections (node:internal/process/task_queues:96:5) {
  errno: -2,
  syscall: 'open',
  code: 'ENOENT',
  path: './oil_and_gas/files/VIIRS_Global_flaring_d.7_slope_0.029353_2017_web_v1.xlsx'
}
npm ERR! Lifecycle script `dataLoader` failed with error: 
npm ERR! Error: command failed 
npm ERR!   in workspace: @blockchain-carbon-accounting/data-postgres@1.0.0 
npm ERR!   at location: /Users/sichen/hyperledger/blockchain-carbon-accounting/data 

> @blockchain-carbon-accounting/data-postgres@1.0.0 dataLoader
> ts-node src/dataLoader.ts "load_product_data" "./oil_and_gas/files/PET_CRD_CRPDN_ADC_MBBL_M.xls" "Data 1" "--format" "EIA" "--source" "https://www.eia.gov/dnav/ng/xls/PET_CRD_CRPDN_ADC_MBBL_M.xls" "--name" "Crude Oil" "--type" "Field Production" "--unit" "Thousand Barrels" "--skip_rows" "2"

=== Starting load_product_data ...
Loading into PostgresDB |████████████████████████████████████████| 100% | ETA: 0s | 0/0 records
=== Loaded ./oil_and_gas/files/PET_CRD_CRPDN_ADC_MBBL_M.xls | Data 1: 0 rows loaded, 0 ignored.
=== Done, we now have 17496 product entries in the DB

> @blockchain-carbon-accounting/data-postgres@1.0.0 dataLoader
> ts-node src/dataLoader.ts "load_product_data" "./oil_and_gas/files/NG_PROD_SUM_A_EPG0_VGM_MMCF_M.xls" "Data 1" "--format" "EIA" "--source" "https://www.eia.gov/dnav/ng/xls/NG_PROD_SUM_A_EPG0_VGM_MMCF_M.xls" "--name" "Natural Gas" "--type" "Marketed Production" "--unit" "MMcf" "--skip_rows" "2"

=== Starting load_product_data ...
Parsing worksheet |████████████████████████████████████████| 100% || 7853/7853
Loading into PostgresDB |████████████████████████████████████████| 100% | ETA: 0s | 4049/593 records
=== Loaded ./oil_and_gas/files/NG_PROD_SUM_A_EPG0_VGM_MMCF_M.xls | Data 1: 593 rows loaded, 3456 ignored.
 - ignored 3456 rows because: Undefined amount
=== Done, we now have 24714 product entries in the DB

> @blockchain-carbon-accounting/data-postgres@1.0.0 dataLoader
> ts-node src/dataLoader.ts "load_product_data" "./oil_and_gas/files/NG_PROD_SUM_A_EPG0_VGM_MMCF_M.xls" "Data 2" "--format" "EIA" "--source" "https://www.eia.gov/dnav/ng/xls/NG_PROD_SUM_A_EPG0_VGM_MMCF_M.xls" "--name" "Natural Gas" "--type" "Marketed Production" "--unit" "MMcf" "--skip_rows" "2"

=== Starting load_product_data ...
Parsing worksheet |████████████████████████████████████████| 100% || 7058/7058
Loading into PostgresDB |████████████████████████████████████████| 100% | ETA: 0s | 2051/401 records
=== Loaded ./oil_and_gas/files/NG_PROD_SUM_A_EPG0_VGM_MMCF_M.xls | Data 2: 401 rows loaded, 1650 ignored.
 - ignored 1650 rows because: Undefined amount
=== Done, we now have 30683 product entries in the DB

> @blockchain-carbon-accounting/data-postgres@1.0.0 dataLoader
> ts-node src/dataLoader.ts "load_product_data" "./oil_and_gas/files/NG_PROD_SUM_A_EPG0_VGV_MMCF_M.xls" "Data 1" "--format" "EIA" "--source" "https://www.eia.gov/dnav/ng/xls/NG_PROD_SUM_A_EPG0_VGV_MMCF_M.xls" "--name" "Natural Gas" "--type" "Vented and Flared" "--unit" "MMcf" "--skip_rows" "2"

=== Starting load_product_data ...
Parsing worksheet |████████████████████████████████████████| 100% || 6690/6690
Loading into PostgresDB |████████████████████████████████████████| 100% | ETA: 0s | 7165/593 records
=== Loaded ./oil_and_gas/files/NG_PROD_SUM_A_EPG0_VGV_MMCF_M.xls | Data 1: 593 rows loaded, 6572 ignored.
 - ignored 6572 rows because: Undefined amount
=== Done, we now have 34785 product entries in the DB

> @blockchain-carbon-accounting/data-postgres@1.0.0 dataLoader
> ts-node src/dataLoader.ts "load_product_data" "./oil_and_gas/files/NG_PROD_SUM_A_EPG0_VGV_MMCF_M.xls" "Data 2" "--format" "EIA" "--source" "https://www.eia.gov/dnav/ng/xls/NG_PROD_SUM_A_EPG0_VGV_MMCF_M.xls" "--name" "Natural Gas" "--type" "Vented and Flared" "--unit" "MMcf" "--skip_rows" "2"

=== Starting load_product_data ...
Parsing worksheet |████████████████████████████████████████| 100% || 6483/6483
Loading into PostgresDB |████████████████████████████████████████| 100% | ETA: 0s | 6337/377 records
=== Loaded ./oil_and_gas/files/NG_PROD_SUM_A_EPG0_VGV_MMCF_M.xls | Data 2: 377 rows loaded, 5960 ignored.
 - ignored 5960 rows because: Undefined amount
=== Done, we now have 35988 product entries in the DB

> @blockchain-carbon-accounting/data-postgres@1.0.0 dataLoader
> ts-node src/dataLoader.ts "load_product_data" "./oil_and_gas/files/NG_PROD_SUM_A_EPG0_VGQ_MMCF_M.xls" "Data 1" "--format" "EIA" "--source" "https://www.eia.gov/dnav/ng/xls/NG_PROD_SUM_A_EPG0_VGQ_MMCF_M.xls" "--name" "Natural Gas" "--type" "Repressuring" "--unit" "MMcf" "--skip_rows" "2"

=== Starting load_product_data ...
Parsing worksheet |████████████████████████████████████████| 100% || 6920/6920
Loading into PostgresDB |████████████████████████████████████████| 100% | ETA: 0s | 7536/593 records
=== Loaded ./oil_and_gas/files/NG_PROD_SUM_A_EPG0_VGQ_MMCF_M.xls | Data 1: 593 rows loaded, 6943 ignored.
 - ignored 6943 rows because: Undefined amount
=== Done, we now have 39719 product entries in the DB

> @blockchain-carbon-accounting/data-postgres@1.0.0 dataLoader
> ts-node src/dataLoader.ts "load_product_data" "./oil_and_gas/files/NG_PROD_SUM_A_EPG0_VGQ_MMCF_M.xls" "Data 2" "--format" "EIA" "--source" "https://www.eia.gov/dnav/ng/xls/NG_PROD_SUM_A_EPG0_VGQ_MMCF_M.xls" "--name" "Natural Gas" "--type" "Repressuring" "--unit" "MMcf" "--skip_rows" "2"

=== Starting load_product_data ...
Parsing worksheet |████████████████████████████████████████| 100% || 6624/6624
Loading into PostgresDB |████████████████████████████████████████| 100% | ETA: 0s | 6369/377 records
=== Loaded ./oil_and_gas/files/NG_PROD_SUM_A_EPG0_VGQ_MMCF_M.xls | Data 2: 377 rows loaded, 5992 ignored.
 - ignored 5992 rows because: Undefined amount
=== Done, we now have 40890 product entries in the DB

> @blockchain-carbon-accounting/data-postgres@1.0.0 dataLoader
> ts-node src/dataLoader.ts "load_product_data" "./oil_and_gas/files/NG_PROD_SUM_A_EPG0_VG9_MMCF_M.xls" "Data 1" "--format" "EIA" "--source" "https://www.eia.gov/dnav/ng/xls/NG_PROD_SUM_A_EPG0_VG9_MMCF_M.xls" "--name" "Natural Gas Plant Liquids" "--type" "Production" "--unit" "Million Cubic Feet" "--skip_rows" "2"

=== Starting load_product_data ...
Parsing worksheet |████████████████████████████████████████| 100% || 5514/5514
Loading into PostgresDB |████████████████████████████████████████| 100% | ETA: 0s | 6868/593 records
=== Loaded ./oil_and_gas/files/NG_PROD_SUM_A_EPG0_VG9_MMCF_M.xls | Data 1: 593 rows loaded, 6275 ignored.
 - ignored 6275 rows because: Undefined amount
=== Done, we now have 45289 product entries in the DB

> @blockchain-carbon-accounting/data-postgres@1.0.0 dataLoader
> ts-node src/dataLoader.ts "load_product_data" "./oil_and_gas/files/NG_PROD_SUM_A_EPG0_VG9_MMCF_M.xls" "Data 2" "--format" "EIA" "--source" "https://www.eia.gov/dnav/ng/xls/NG_PROD_SUM_A_EPG0_VG9_MMCF_M.xls" "--name" "Natural Gas Plant Liquids" "--type" "Production" "--unit" "Million Cubic Feet" "--skip_rows" "2"

=== Starting load_product_data ...
Parsing worksheet |████████████████████████████████████████| 100% || 2029/2029
Loading into PostgresDB |████████████████████████████████████████| 100% | ETA: 0s | 4540/286 records
=== Loaded ./oil_and_gas/files/NG_PROD_SUM_A_EPG0_VG9_MMCF_M.xls | Data 2: 286 rows loaded, 4254 ignored.
 - ignored 4254 rows because: Undefined amount
=== Done, we now have 46469 product entries in the DB

> @blockchain-carbon-accounting/data-postgres@1.0.0 dataLoader
> ts-node src/dataLoader.ts "load_product_data" "./oil_and_gas/files/NG_PROD_SUM_A_EPG0_FGW_MMCF_M.xls" "Data 1" "--format" "EIA" "--source" "https://www.eia.gov/dnav/ng/xls/NG_PROD_SUM_A_EPG0_FGW_MMCF_M.xls" "--name" "Natural Gas" "--type" "Gross Withdrawals" "--unit" "MMcf" "--skip_rows" "2"

=== Starting load_product_data ...
Parsing worksheet |████████████████████████████████████████| 100% || 7481/7481
Loading into PostgresDB |████████████████████████████████████████| 100% | ETA: 0s | 4421/593 records
=== Loaded ./oil_and_gas/files/NG_PROD_SUM_A_EPG0_FGW_MMCF_M.xls | Data 1: 593 rows loaded, 3828 ignored.
 - ignored 3828 rows because: Undefined amount
=== Done, we now have 53315 product entries in the DB

> @blockchain-carbon-accounting/data-postgres@1.0.0 dataLoader
> ts-node src/dataLoader.ts "load_product_data" "./oil_and_gas/files/NG_PROD_SUM_A_EPG0_FGW_MMCF_M.xls" "Data 2" "--format" "EIA" "--source" "https://www.eia.gov/dnav/ng/xls/NG_PROD_SUM_A_EPG0_FGW_MMCF_M.xls" "--name" "Natural Gas" "--type" "Gross Withdrawals" "--unit" "MMcf" "--skip_rows" "2"

=== Starting load_product_data ...
Parsing worksheet |████████████████████████████████████████| 100% || 6788/6788
Loading into PostgresDB |████████████████████████████████████████| 100% | ETA: 0s | 1682/377 records
=== Loaded ./oil_and_gas/files/NG_PROD_SUM_A_EPG0_FGW_MMCF_M.xls | Data 2: 377 rows loaded, 1305 ignored.
 - ignored 1305 rows because: Undefined amount
=== Done, we now have 59173 product entries in the DB

> @blockchain-carbon-accounting/data-postgres@1.0.0 dataLoader
> ts-node src/dataLoader.ts "load_product_data" "./oil_and_gas/files/flaring_monitor_detailed_observations.csv" "Sheet1" "--format" "FlareMonitor" "--source" "https://raw.githubusercontent.com/flaringmonitor/viirs-flare-data/main/processed/flaring_monitor_detailed_observations.csv" "--name" "Methane" "--type" "Flaring" "--unit" "MMcf"

=== Starting load_product_data ...
Parsing worksheet |████████████████████████████████████████| 100% || 2761458/2761458
Loading into PostgresDB |████████████████████████████████████████| 100% | ETA: 0s | 287132/287132 records
=== Loaded ./oil_and_gas/files/flaring_monitor_detailed_observations.csv | Sheet1: 85198 rows loaded, 201934 ignored.
 - ignored 201934 rows because: No row identifier
=== Done, we now have 144371 product entries in the DB

> @blockchain-carbon-accounting/data-postgres@1.0.0 dataLoader
> ts-node src/dataLoader.ts "load_product_data" "./oil_and_gas/files/flaring_monitor_company_stats_satellite_modeled.csv" "Sheet1" "--format" "FlareMonitor" "--source" "https://raw.githubusercontent.com/flaringmonitor/viirs-flare-data/main/processed/flaring_monitor_company_stats_satellite_modeled.csv"

=== Starting load_product_data ...
Parsing worksheet |████████████████████████████████████████| 100% || 21852/21852
Loading into PostgresDB |████████████████████████████████████████| 100% | ETA: 0s | 20039/20039 records
=== Loaded ./oil_and_gas/files/flaring_monitor_company_stats_satellite_modeled.csv | Sheet1: 17672 rows loaded, 2367 ignored.
 - ignored 2367 rows because: Undefined col amount
=== Done, we now have 162043 product entries in the DB

> @blockchain-carbon-accounting/data-postgres@1.0.0 dataLoader
> ts-node src/dataLoader.ts "load_product_data" "./oil_and_gas/files/flaring_monitor_company_stats_reported.csv" "Sheet1" "--format" "FlareMonitor" "--source" "https://raw.githubusercontent.com/flaringmonitor/viirs-flare-data/main/processed/flaring_monitor_company_stats_reported.csv"

=== Starting load_product_data ...
Parsing worksheet |████████████████████████████████████████| 100% || 186041/186041
Loading into PostgresDB |████████████████████████████████████████| 100% | ETA: 0s | 141404/141404 records
=== Loaded ./oil_and_gas/files/flaring_monitor_company_stats_reported.csv | Sheet1: 50230 rows loaded, 91174 ignored.
 - ignored 91174 rows because: Undefined col amount
=== Done, we now have 212273 product entries in the DB

> @blockchain-carbon-accounting/data-postgres@1.0.0 dataLoader
> ts-node src/dataLoader.ts "load_product_data" "./oil_and_gas/files/flaring_monitor_basin_stats.csv" "Sheet1" "--format" "FlareMonitor" "--source" "https://raw.githubusercontent.com/flaringmonitor/viirs-flare-data/main/processed/flaring_monitor_basin_stats.csv"

=== Starting load_product_data ...
Parsing worksheet |████████████████████████████████████████| 100% || 1594/1594
Loading into PostgresDB |████████████████████████████████████████| 100% | ETA: 0s | 1392/1392 records
=== Loaded ./oil_and_gas/files/flaring_monitor_basin_stats.csv | Sheet1: 1368 rows loaded, 24 ignored.
 - ignored 24 rows because: Undefined col amount
=== Done, we now have 213641 product entries in the DB

> @blockchain-carbon-accounting/data-postgres@1.0.0 dataLoader
> ts-node src/dataLoader.ts "load_og_assets" "./oil_and_gas/files/Oil_and_Natural_Gas_Wells.geojson" "--format" "US_asset_data" "--source" "https://hifld-geoplatform.opendata.arcgis.com/datasets/geoplatform::oil-and-natural-gas-wells/explore"

=== Starting load_og_assets ...
Loading into PostgresDB |░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░| 0% | ETA: 0s | 0/1506238 records[Error: ENOENT: no such file or directory, open '././oil_and_gas/files/Oil_and_Natural_Gas_Wells.geojson'
  errno: -2,
  code: 'ENOENT',
  syscall: 'open',
  path: '././oil_and_gas/files/Oil_and_Natural_Gas_Wells.geojson'
}
npm ERR! Lifecycle script `dataLoader` failed with error: 
npm ERR! Error: command failed 
npm ERR!   in workspace: @blockchain-carbon-accounting/data-postgres@1.0.0 
npm ERR!   at location: /Users/sichen/hyperledger/blockchain-carbon-accounting/data

I see a table product in my database with data, oil_and_gas_asset which is empty, and no other new tables.

brioux commented 2 years ago

@sichen1234

The errors reported above are because the download.sh script does not yet work for the following files.

  1. All the VIIRS_Global_flaring_d. (https://eogdata.mines.edu/global_flare_data requires free subscription)
  2. Oil_and_Natural_Gas_Wells.geojson available here. This file contains all the data for oil_and_gas_asset, this is why your are getting an empty table

These (and all other files) can be downloaded manually from the google drive I set up here

Need to revise the download.sh script to work for large files stored on google drive, or setup another link to download these with curl.

sichen1234 commented 2 years ago

This is working now. I updated the README.md in data/oil_and_gas/ a little bit.