Closed SeijiSensei closed 4 years ago
I'm not sure what you're trying to do. I think perhaps you misunderstand what yamltodb
can be used for. The input file to yamltodb
is (usually) a YAML file created by dbtoyaml
. It has a specific syntax (see examples at https://pyrseas.readthedocs.io/en/latest/table.html, albeit shown mostly using JSON) that describes schemas, tables and other objects in a Postgres database, so that it can compare it to the target database. The sample.yaml.txt
(and the full legislators-current.yaml
) do not adhere to that syntax.
Ah, I thought I could feed it a YAML file and have yamltodb parse the structure of that file and create the SQL commands to populate the database. Do you know of any utilities that can do that? Probably not would be my guess.
If you don't have suggestions for other utilities, you can close this comment.
Thanks!
I'm afraid I haven't heard of anything that would help directly with the YAML. The problem is that the file is hierarchical (and with lots of redundancy) and Postgres is, for the most part, relational. At first glance, there are two main "tables": the legislators (from the id
to the bio: gender
in the sample) and the terms (the repeating groups under terms
). It wouldn't be too difficult to whip up a program in your favorite language to load that.
IIRC Dimitri Fontaine had written some articles (now also a book) and a utility (pgloader) that could help, but AFAIK you'd first have to massage the data into CSV (that's available on that github repo) or fixed format. Postgres also supports JSON formats so in theory, you could load the file into a single column "table" and then manipulate with PG's mixture of SQL and JSON. Or if you go with the CSV format, you can load it directly (without the terms) into a "normal" table but you'd first have to define it (open the CSV in an editor or spreadsheet program and the first row gives you the column names/headings).
$ yamltodb legislators legislators-current.yaml Traceback (most recent call last): File "/usr/local/bin/yamltodb", line 10, in
sys.exit(main())
File "/usr/local/lib/python3.7/dist-packages/pyrseas/yamltodb.py", line 50, in main
stmts = db.diff_map(inmap)
File "/usr/local/lib/python3.7/dist-packages/pyrseas/database.py", line 526, in diff_map
self.from_map(input_map, langs)
File "/usr/local/lib/python3.7/dist-packages/pyrseas/database.py", line 375, in from_map
if key.startswith('schema '):
AttributeError: 'dict' object has no attribute 'startswith'
Kubuntu 19.10 Postgresql 11.5 Python3: 3.7.5-1 Python3-pip: 18.1-5 PyYaml: 5.3
Sample entry of data
sample.yaml.txt
from https://github.com/unitedstates/congress-legislators/blob/master/legislators-current.yaml