Improve integration tests. Add SQLite & requester pays docs

[zaneselvans]

Several changes in this PR:

• Default to the catalog "version" dev -- only hard-code a real version like v2022.06.15 when we tag a release, which will only happen after PUDL tags a release.
• Remove accidental duplicate CODE_OF_CONDUCT file.
• Add some tests for the SQLite catalog entries (do the files actually exist?)
• Add some basic metadata tests (do all catalog entries have the minimal required fields like license?)
• Break out the bucket name and version string as separate constants for re-use in various contexts.
• Add usage examples to the notebook for SQLite catalog entries
• Improve documentation on setting up requester pays for new users.

Closes #25 Closes #29 Closes #30

[codecov[bot]]

Codecov Report

Merging #32 (8a9e3b1) into main (ddf73a3) will not change coverage. The diff coverage is 100.0%.

@@           Coverage Diff           @@
##             main      #32   +/-   ##
=======================================
  Coverage   100.0%   100.0%           
=======================================
  Files           2        2           
  Lines          42       44    +2     
=======================================
+ Hits           42       44    +2     

Impacted Files	Coverage Δ
src/pudl_catalog/__init__.py	100.0% <100.0%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update fbe9ab9...8a9e3b1. Read the comment docs.

[katie-lamb]

I might have a unique situation because of my previous use of GCP but this is the problem I ran into.

First I tried to follow the docs in order and clicked on Create Project:

• When I clicked on Create Project and tried to make a new project I was required to fill in Billing Account info and Location (of the billing account I think), not just project name as is suggested in the Catalyst docs
• I then clicked on the Billing Account drop down menu and had to choose from existing billing accounts, which was just the Catalyst Cooperative account associated with DBCP. The Location field also was a drop down and I had to choose the dbcp-dev folder
• When I tried to create a project with these specifications I got an error. I think this is because I don't have appropriate permissions with DBCP.

I then tried to create a new billing account that isn't associated with DBCP:

• I went to https://console.cloud.google.com/billing and clicked on Create Account
• I filled in the fields but then got this message when I tried to create the account: You must have Billing Account Creator permissions for your organization in order to create a new billing account. Your Organization Administrator can assign this role to you.

Finally I tried going to the IAM admin page:

• On the main IAM tab I got the message: To view this page, select a project.
• Now I'm stuck because I can't create a project due to the billing account issues

This seems like a classic GCP issue. I think to make a project I need to be given permissions to create billing accounts in the Catalyst org.

From a docs perspective, what's maybe the biggest discrepancy is that the Create Project page only requires you to enter a project name and for me I need to add billing account info. Not sure why this is. It seems hard to make docs to account for every scenario but we should probably try to sort this out a little. Maybe the section for creating a billing account should come before the project creation? Again, this might be weird for me because I have interacted with GCP before and I guess someone with a little experience with GCP probably has more know how on how to create projects and the Catalyst instructions are less essential. Maybe someone with a personal email that's never interacted with GCP before should try this and see what happens.

[zaneselvans]

Hey @katie-lamb would you be willing to try working through the instructions with your own gmail account rather than Catalyst? Since that's the way it'll probably work for an outside person?

[katie-lamb]

Okay so I've now tried to do this from another non-Catalyst email. It's my alumni email that has never interacted with GCP before. It shouldn't matter that it's not a gmail account I think. Here are some discrepancies from the docs:

• When I created a GCP account (after clicking "Get started for free") it made me set up a billing account
• When I created a new project I had to associate a billing account and location with it (in the Catalyst docs it says you just need to provide it project name)
• When I went to the IAM & Admin page I needed to select the new project from the top drop down menu. At first it defaulted to No organization and I didn't have permission to view the page.
• When editing permissions in the IAM & Admin page I had to first select "Add Another Role" so I didn't remove myself as owner of the project
• After gcloud auth login it prompted me to set the project with gcloud config set project MY_PROJECT_ID which gave me the message Updated property [core/project]. It's not clear in the docs if I should immediately run gcloud init after logging in (see my comment on this line in the docs).
• After running gcloud init and re-initializing the configuration with new settings as well as establishing application default credentials I ran gsutil ls gs://intake.catalyst.coop and got the message BadRequestException: 400 Bucket is a requester pays bucket but no user project provided.. The project id I just created is the default project, so I'm not entirely surewhat's wrong here.

I can also try from my personal gmail - which was interacted with GCP before. Or I can create a fresh gmail account just for this but I don't think it should matter if it's a gmail account or other org.

[zaneselvans]

And your Stanford Alumni account allowed you to create projects & billing accounts? That seems a little bit surprising to me.

I was not sure what order the gcloud init and gcloud auth login stuff was supposed to happen in.

At the end it's complaining because it doesn't know where to bill the access costs to. So either it doesn't know what project to bill to, or the project it's trying to bill to has no billing account. Does it work if you explicitly pass it the billing project like this:

gsutil -u your_new_project_name ls gs://intake.catalyst.coop

If it does, then the project has billing associated with it, but that project isn't being used by default. If it doesn't work, then that project doesn't have billing (or you don't have permission to spend).

[review-notebook-app[bot]]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[zaneselvans]

Hey yes, only the database / parquet file that's being accessed is downloaded, one by one, so it won't grab the ferc1 DB until you touch it, and it won't grab the pudl DB until you touch it, so I think that's the expected behavior.