search

hyperledger-labs / blockchain-carbon-accounting

This project implements blockchain applications for climate action and accounting, including emissions calculations, carbon trading, and validation of climate claims. It is part of the Linux Foundation's Hyperledger Climate Action and Accounting SIG.
https://wiki.hyperledger.org/display/CASIG/Carbon+Accounting+and+Certification+WG
Apache License 2.0
195 stars 104 forks source link

problem loading oil and gas data #666

Closed sichen1234 closed 1 year ago

sichen1234 commented 1 year ago

I tried data/loadOGdata.sh but got the attached errors.

loadOGdata_error.log.zip

brioux commented 1 year ago

@sichen1234 If running from the project root you can use 

npm run loadSeeds:OG

which will execute loadOGdata.sh from the /data workspace.

calling data/loadOGdata.sh will not work as the dataLoader script is not visible from project root

sichen1234 commented 1 year ago

Thanks, but when I ran it, it just kept running like this forever.

Screen Shot 2023-02-24 at 7 16 50 AM

Eventually it crashed my computer.

Is there a log that could show me what's going on?

brioux commented 1 year ago

@sichen1234 Looks like an issue when pulling oil and gas asset data from Oil_and_Natural_Gas_Wells.geojson ...

Made some changes in PR 674 that might resolve your issue.. The asset data is no longer loaded by default in loadOGdata.sh and the Readme have been modified

You can load the asset data manually with 

cd data/
npm run dataLoader load_og_assets "./oil_and_gas/files/Oil_and_Natural_Gas_Wells.geojson" -- --format US_asset_data --source "https://hifld-geoplatform.opendata.arcgis.com/datasets/geoplatform::oil-and-natural-gas-wells/explore"

It can take a long time to load the +1.5 million assets. This might have caused the crash... If still facing issue with the above cmd I added binary files to download.sh script to avoid this long loading task. The files can be used to seed/restore the following tables using postgres admin console (after setting up the DB schema with npm run pg:init):

  1. oil_and_gas_asset
  2. operator
  3. asset_operator

After populating these tables sh loadOGdata.sh should behave as expected.

sichen1234 commented 1 year ago

The problem was that ts-node needed to be installed first with

npm install -g ts-node

After that it is loading but very slowly:

=== Starting load_og_assets ...
Loading into PostgresDB |░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░| 0% | ETA: 764645s | 4318/1506238 records