search

ipfs-inactive / archives

[ARCHIVED] Repo to coordinate archival efforts with IPFS
https://awesome.ipfs.io/datasets
183 stars 24 forks source link

MDSConnect raw data (Library of Congress) #152

Open ghost opened 7 years ago

ghost commented 7 years ago

The Library of Congress announced today that it is making 25 million records in its online catalog available for free bulk download at loc.gov/cds/products/marcDist.php. This is the largest release of digital records in the Library’s history. -- https://www.loc.gov/item/prn-17-068?loclr=twloc

I grabbed a raw dump of the files and added it on the storage host nihal (QmepgFW7BHEtU4pZJdxaNiv75mKLLRQnPi1KaaXmQN4V1a)

> screen bash -c "wget --mirror --convert-links --no-parent -nv https://loc.gov/cds/downloads/MDSConnect/ 2>&1 | tee loc.log"
> ipfs add -r loc.gov/cds/downloads/MDSConnect/
added QmWSzgkftVrkh2859bGT44ahzoqcGhFkjsrQUtHen9hVw9 loc.gov/cds/downloads/MDSConnect

Result: https://ipfs.io/ipfs/QmWSzgkftVrkh2859bGT44ahzoqcGhFkjsrQUtHen9hVw9/ https://ipfs.io/ipfs/QmcvfB6pAqUfTnuAK8zFKVxbdhopnBPveJrDcy1JAA7HX5

ghost commented 7 years ago

It's slightly more than 100 GB:

> ipfs object stat QmWSzgkftVrkh2859bGT44ahzoqcGhFkjsrQUtHen9hVw9/
NumLinks: 301
BlockSize: 21623
LinksSize: 21621
DataSize: 2
CumulativeSize: 105535285885
ghost commented 7 years ago

I'm only seeing 18GB after running that same wget command. Any ideas?

ghost commented 7 years ago

I don't know :) Here's what I downloaded:

> du -sh loc.gov/cds/downloads/MDSConnect/
99G loc.gov/cds/downloads/MDSConnect/
> ls -la loc.gov/cds/downloads/MDSConnect/
total 103037888
drwxr-xr-x 1 lars users       15944 May 17 23:03 .
drwxr-xr-x 1 lars users          20 May 17 03:31 ..
-rw-r--r-- 1 lars users    69715040 Mar  6 18:23 BooksAll.2014.part01.marc8.gz
-rw-r--r-- 1 lars users   241298365 Mar  6 18:25 BooksAll.2014.part01.utf8.gz
-rw-r--r-- 1 lars users   699767853 Mar  6 18:26 BooksAll.2014.part01.xml.gz
-rw-r--r-- 1 lars users   203707455 Mar  6 18:27 BooksAll.2014.part02.marc8.gz
-rw-r--r-- 1 lars users   203934030 Mar  6 18:29 BooksAll.2014.part02.utf8.gz
-rw-r--r-- 1 lars users   574933045 Mar  6 18:30 BooksAll.2014.part02.xml.gz
-rw-r--r-- 1 lars users   196930346 Mar  6 18:31 BooksAll.2014.part03.marc8.gz
-rw-r--r-- 1 lars users   197207496 Mar  6 18:32 BooksAll.2014.part03.utf8.gz
-rw-r--r-- 1 lars users   572514807 Mar  6 18:33 BooksAll.2014.part03.xml.gz
-rw-r--r-- 1 lars users   181828576 Mar  6 18:34 BooksAll.2014.part04.marc8.gz
-rw-r--r-- 1 lars users   182253854 Mar  6 18:35 BooksAll.2014.part04.utf8.gz
-rw-r--r-- 1 lars users   545114876 Mar  6 18:36 BooksAll.2014.part04.xml.gz
-rw-r--r-- 1 lars users   173423594 Mar  6 18:37 BooksAll.2014.part05.marc8.gz
-rw-r--r-- 1 lars users   173986807 Mar  6 18:38 BooksAll.2014.part05.utf8.gz
-rw-r--r-- 1 lars users   529003733 Mar  6 18:39 BooksAll.2014.part05.xml.gz
-rw-r--r-- 1 lars users   182129529 Mar  6 18:40 BooksAll.2014.part06.marc8.gz
-rw-r--r-- 1 lars users   182671671 Mar  6 18:41 BooksAll.2014.part06.utf8.gz
-rw-r--r-- 1 lars users   542825762 Mar  6 18:43 BooksAll.2014.part06.xml.gz
-rw-r--r-- 1 lars users   187315378 Mar  6 18:44 BooksAll.2014.part07.marc8.gz
-rw-r--r-- 1 lars users   187912394 Mar  6 18:45 BooksAll.2014.part07.utf8.gz
-rw-r--r-- 1 lars users   555739812 Mar  6 18:46 BooksAll.2014.part07.xml.gz
-rw-r--r-- 1 lars users   189680872 Mar  6 18:47 BooksAll.2014.part08.marc8.gz
-rw-r--r-- 1 lars users   190342088 Mar  6 18:49 BooksAll.2014.part08.utf8.gz
-rw-r--r-- 1 lars users   562044482 Mar  6 18:50 BooksAll.2014.part08.xml.gz
-rw-r--r-- 1 lars users   193086744 Mar  6 18:51 BooksAll.2014.part09.marc8.gz
-rw-r--r-- 1 lars users   193790793 Mar  6 18:52 BooksAll.2014.part09.utf8.gz
-rw-r--r-- 1 lars users   563448752 Mar  6 18:54 BooksAll.2014.part09.xml.gz
-rw-r--r-- 1 lars users   195052177 Mar  6 18:55 BooksAll.2014.part10.marc8.gz
-rw-r--r-- 1 lars users   195750552 Mar  6 18:56 BooksAll.2014.part10.utf8.gz
-rw-r--r-- 1 lars users   571278098 Mar  6 18:57 BooksAll.2014.part10.xml.gz
-rw-r--r-- 1 lars users   195386784 Mar  6 18:59 BooksAll.2014.part11.marc8.gz
-rw-r--r-- 1 lars users   196300104 Mar  6 19:00 BooksAll.2014.part11.utf8.gz
-rw-r--r-- 1 lars users   571091283 Mar  6 19:01 BooksAll.2014.part11.xml.gz
-rw-r--r-- 1 lars users   194946672 Mar  6 19:03 BooksAll.2014.part12.marc8.gz
-rw-r--r-- 1 lars users   195849864 Mar  6 19:04 BooksAll.2014.part12.utf8.gz
-rw-r--r-- 1 lars users   570395829 Mar  6 19:05 BooksAll.2014.part12.xml.gz
-rw-r--r-- 1 lars users   203152504 Mar  6 19:07 BooksAll.2014.part13.marc8.gz
-rw-r--r-- 1 lars users   204472578 Mar  6 19:08 BooksAll.2014.part13.utf8.gz
-rw-r--r-- 1 lars users   592712132 Mar  6 19:09 BooksAll.2014.part13.xml.gz
-rw-r--r-- 1 lars users   200994841 Mar  6 19:11 BooksAll.2014.part14.marc8.gz
-rw-r--r-- 1 lars users   202046295 Mar  6 19:12 BooksAll.2014.part14.utf8.gz
-rw-r--r-- 1 lars users   583943437 Mar  6 19:13 BooksAll.2014.part14.xml.gz
-rw-r--r-- 1 lars users   203542905 Mar  6 19:15 BooksAll.2014.part15.marc8.gz
-rw-r--r-- 1 lars users   204589201 Mar  6 19:16 BooksAll.2014.part15.utf8.gz
-rw-r--r-- 1 lars users   590033819 Mar  6 19:17 BooksAll.2014.part15.xml.gz
-rw-r--r-- 1 lars users   221243670 Mar  6 19:19 BooksAll.2014.part16.marc8.gz
-rw-r--r-- 1 lars users   221115040 Mar  6 19:20 BooksAll.2014.part16.utf8.gz
-rw-r--r-- 1 lars users   641499947 Mar  6 19:22 BooksAll.2014.part16.xml.gz
-rw-r--r-- 1 lars users   214644431 Mar  6 19:23 BooksAll.2014.part17.marc8.gz
-rw-r--r-- 1 lars users   214918525 Mar  6 19:25 BooksAll.2014.part17.utf8.gz
-rw-r--r-- 1 lars users   618703199 Mar  6 19:26 BooksAll.2014.part17.xml.gz
-rw-r--r-- 1 lars users   224531239 Mar  6 19:27 BooksAll.2014.part18.marc8.gz
-rw-r--r-- 1 lars users   224156107 Mar  6 19:29 BooksAll.2014.part18.utf8.gz
-rw-r--r-- 1 lars users   456765866 May 17 05:59 BooksAll.2014.part18.xml.gz
-rw-r--r-- 1 lars users    71469761 Mar  6 19:32 BooksAll.2014.part19.marc8.gz
-rw-r--r-- 1 lars users   226774478 Mar  6 19:33 BooksAll.2014.part19.utf8.gz
-rw-r--r-- 1 lars users   659834069 Mar  6 19:35 BooksAll.2014.part19.xml.gz
-rw-r--r-- 1 lars users   222810801 Mar  6 19:36 BooksAll.2014.part20.marc8.gz
-rw-r--r-- 1 lars users   222901634 Mar  6 19:37 BooksAll.2014.part20.utf8.gz
-rw-r--r-- 1 lars users   646174224 Mar  6 19:39 BooksAll.2014.part20.xml.gz
-rw-r--r-- 1 lars users   228088426 Mar  6 19:40 BooksAll.2014.part21.marc8.gz
-rw-r--r-- 1 lars users   227911669 Mar  6 19:41 BooksAll.2014.part21.utf8.gz
-rw-r--r-- 1 lars users   665956191 Mar  6 19:43 BooksAll.2014.part21.xml.gz
-rw-r--r-- 1 lars users   225335764 Mar  6 19:44 BooksAll.2014.part22.marc8.gz
-rw-r--r-- 1 lars users   225286411 Mar  6 19:45 BooksAll.2014.part22.utf8.gz
-rw-r--r-- 1 lars users   657160123 Mar  6 19:47 BooksAll.2014.part22.xml.gz
-rw-r--r-- 1 lars users   228905706 Mar  6 19:48 BooksAll.2014.part23.marc8.gz
-rw-r--r-- 1 lars users   228945550 Mar  6 19:50 BooksAll.2014.part23.utf8.gz
-rw-r--r-- 1 lars users   668859353 Mar  6 19:51 BooksAll.2014.part23.xml.gz
-rw-r--r-- 1 lars users   231010312 Mar  6 19:53 BooksAll.2014.part24.marc8.gz
-rw-r--r-- 1 lars users   231475659 Mar  6 19:54 BooksAll.2014.part24.utf8.gz
-rw-r--r-- 1 lars users   671986563 Mar  6 19:56 BooksAll.2014.part24.xml.gz
-rw-r--r-- 1 lars users   241602985 Mar  6 19:57 BooksAll.2014.part25.marc8.gz
-rw-r--r-- 1 lars users   240955227 Mar  6 19:59 BooksAll.2014.part25.utf8.gz
-rw-r--r-- 1 lars users   703426547 Mar  6 20:00 BooksAll.2014.part25.xml.gz
-rw-r--r-- 1 lars users   237474386 Mar  6 20:01 BooksAll.2014.part26.marc8.gz
-rw-r--r-- 1 lars users   238178024 Mar  6 20:03 BooksAll.2014.part26.utf8.gz
-rw-r--r-- 1 lars users   688003265 Mar  6 20:04 BooksAll.2014.part26.xml.gz
-rw-r--r-- 1 lars users   237290429 Mar  6 20:06 BooksAll.2014.part27.marc8.gz
-rw-r--r-- 1 lars users   237710635 Mar  6 20:07 BooksAll.2014.part27.utf8.gz
-rw-r--r-- 1 lars users   690960392 Mar  6 20:08 BooksAll.2014.part27.xml.gz
-rw-r--r-- 1 lars users   241833691 Mar  6 20:10 BooksAll.2014.part28.marc8.gz
-rw-r--r-- 1 lars users   242377031 Mar  6 20:11 BooksAll.2014.part28.utf8.gz
-rw-r--r-- 1 lars users   700636950 Mar  6 20:13 BooksAll.2014.part28.xml.gz
-rw-r--r-- 1 lars users   252353493 Mar  6 20:14 BooksAll.2014.part29.marc8.gz
-rw-r--r-- 1 lars users   252763519 Mar  6 20:15 BooksAll.2014.part29.utf8.gz
-rw-r--r-- 1 lars users   724849542 Mar  6 20:17 BooksAll.2014.part29.xml.gz
-rw-r--r-- 1 lars users   254035211 Mar  6 20:18 BooksAll.2014.part30.marc8.gz
-rw-r--r-- 1 lars users   254643556 Mar  6 20:20 BooksAll.2014.part30.utf8.gz
-rw-r--r-- 1 lars users   731563945 Mar  6 20:21 BooksAll.2014.part30.xml.gz
-rw-r--r-- 1 lars users   258590257 Mar  6 20:22 BooksAll.2014.part31.marc8.gz
-rw-r--r-- 1 lars users   258920866 Mar  6 20:24 BooksAll.2014.part31.utf8.gz
-rw-r--r-- 1 lars users   742418454 Mar  6 20:25 BooksAll.2014.part31.xml.gz
-rw-r--r-- 1 lars users   272235356 Mar  6 20:27 BooksAll.2014.part32.marc8.gz
-rw-r--r-- 1 lars users   272071674 Mar  6 20:28 BooksAll.2014.part32.utf8.gz
-rw-r--r-- 1 lars users   774985570 Mar  6 20:30 BooksAll.2014.part32.xml.gz
-rw-r--r-- 1 lars users   278033342 Mar  6 20:31 BooksAll.2014.part33.marc8.gz
-rw-r--r-- 1 lars users   277343192 Mar  6 20:33 BooksAll.2014.part33.utf8.gz
-rw-r--r-- 1 lars users   780125418 Mar  6 20:34 BooksAll.2014.part33.xml.gz
-rw-r--r-- 1 lars users   282031930 Mar  6 20:35 BooksAll.2014.part34.marc8.gz
-rw-r--r-- 1 lars users   280731807 Mar  6 20:37 BooksAll.2014.part34.utf8.gz
-rw-r--r-- 1 lars users   790682376 Mar  6 20:39 BooksAll.2014.part34.xml.gz
-rw-r--r-- 1 lars users   277102937 Mar  6 20:40 BooksAll.2014.part35.marc8.gz
-rw-r--r-- 1 lars users   277720141 Mar  6 20:42 BooksAll.2014.part35.utf8.gz
-rw-r--r-- 1 lars users   776713285 Mar  6 20:44 BooksAll.2014.part35.xml.gz
-rw-r--r-- 1 lars users   288079691 Mar  6 20:45 BooksAll.2014.part36.marc8.gz
-rw-r--r-- 1 lars users   287548806 Mar  6 20:47 BooksAll.2014.part36.utf8.gz
-rw-r--r-- 1 lars users   808790988 Mar  6 20:49 BooksAll.2014.part36.xml.gz
-rw-r--r-- 1 lars users   295612804 Mar  6 20:50 BooksAll.2014.part37.marc8.gz
-rw-r--r-- 1 lars users   295801774 Mar  6 20:52 BooksAll.2014.part37.utf8.gz
-rw-r--r-- 1 lars users   819919343 Mar  6 20:54 BooksAll.2014.part37.xml.gz
-rw-r--r-- 1 lars users   309400161 Mar  6 20:56 BooksAll.2014.part38.marc8.gz
-rw-r--r-- 1 lars users   309675475 Mar  6 20:58 BooksAll.2014.part38.utf8.gz
-rw-r--r-- 1 lars users   850580276 Mar  6 20:59 BooksAll.2014.part38.xml.gz
-rw-r--r-- 1 lars users   305520335 Mar  6 21:01 BooksAll.2014.part39.marc8.gz
-rw-r--r-- 1 lars users   303987823 Mar  6 21:03 BooksAll.2014.part39.utf8.gz
-rw-r--r-- 1 lars users   855291792 Mar  6 21:05 BooksAll.2014.part39.xml.gz
-rw-r--r-- 1 lars users   297834423 Mar  6 21:06 BooksAll.2014.part40.marc8.gz
-rw-r--r-- 1 lars users   297543947 Mar  6 21:08 BooksAll.2014.part40.utf8.gz
-rw-r--r-- 1 lars users   839434664 Mar  6 21:09 BooksAll.2014.part40.xml.gz
-rw-r--r-- 1 lars users    72981354 Mar  6 21:10 BooksAll.2014.part41.marc8.gz
-rw-r--r-- 1 lars users    73301759 Mar  6 21:10 BooksAll.2014.part41.utf8.gz
-rw-r--r-- 1 lars users   214206332 Mar  6 21:10 BooksAll.2014.part41.xml.gz
-rw-r--r-- 1 lars users        1000 Mar  6 22:01 Class.thru.d130228.label.gz
-rw-r--r-- 1 lars users   248527814 Mar  6 22:02 Class.thru.d130228.records.marc8.gz
-rw-r--r-- 1 lars users   248561562 Mar  6 22:02 Class.thru.d130228.records.utf8.gz
-rw-r--r-- 1 lars users   729950005 Mar  6 22:03 Class.thru.d130228.records.xml.gz
-rw-r--r-- 1 lars users    50765920 Mar  6 22:03 Class.thru.d130228.report.gz
-rw-r--r-- 1 lars users    14113402 Mar  6 21:15 Computer.Files.2014.part01.marc8.gz
-rw-r--r-- 1 lars users    14124087 Mar  6 21:15 Computer.Files.2014.part01.utf8.gz
-rw-r--r-- 1 lars users    34579155 Mar  6 21:15 Computer.Files.2014.part01.xml.gz
-rw-r--r-- 1 lars users   343670846 Mar  6 21:16 Maps.2014.part01.marc8.gz
-rw-r--r-- 1 lars users   344230905 Mar  6 21:18 Maps.2014.part01.utf8.gz
-rw-r--r-- 1 lars users   971330020 Mar  6 21:19 Maps.2014.part01.xml.gz
-rw-r--r-- 1 lars users   313887801 Mar  6 21:21 Music.2014.part01.marc8.gz
-rw-r--r-- 1 lars users   314584973 Mar  6 21:23 Music.2014.part01.utf8.gz
-rw-r--r-- 1 lars users   843214097 Mar  6 21:25 Music.2014.part01.xml.gz
-rw-r--r-- 1 lars users   319653660 Mar  6 21:26 Music.2014.part02.marc8.gz
-rw-r--r-- 1 lars users   320326381 Mar  6 21:28 Music.2014.part02.utf8.gz
-rw-r--r-- 1 lars users   853042577 Mar  6 21:30 Music.2014.part02.xml.gz
-rw-r--r-- 1 lars users  4886872448 Mar  6 23:56 Names.2014.combined.marc8.gz
-rw-r--r-- 1 lars users  4911180917 Mar  7 00:24 Names.2014.combined.utf8.gz
-rw-r--r-- 1 lars users 12388696929 Mar  7 00:53 Names.2014.combined.xml.gz
-rw-r--r-- 1 lars users   143881637 Mar  6 22:21 Names.2014.part01.marc8.gz
-rw-r--r-- 1 lars users   144557890 Mar  6 22:22 Names.2014.part01.utf8.gz
-rw-r--r-- 1 lars users   376256048 Mar  6 22:22 Names.2014.part01.xml.gz
-rw-r--r-- 1 lars users   147732769 Mar  6 22:23 Names.2014.part02.marc8.gz
-rw-r--r-- 1 lars users   148369572 Mar  6 22:24 Names.2014.part02.utf8.gz
-rw-r--r-- 1 lars users   372804530 Mar  6 22:25 Names.2014.part02.xml.gz
-rw-r--r-- 1 lars users   146881502 Mar  6 22:25 Names.2014.part03.marc8.gz
-rw-r--r-- 1 lars users   147761163 Mar  6 22:26 Names.2014.part03.utf8.gz
-rw-r--r-- 1 lars users   373275684 Mar  6 22:27 Names.2014.part03.xml.gz
-rw-r--r-- 1 lars users   145373939 Mar  6 22:28 Names.2014.part04.marc8.gz
-rw-r--r-- 1 lars users   146127016 Mar  6 22:28 Names.2014.part04.utf8.gz
-rw-r--r-- 1 lars users   369282192 Mar  6 22:29 Names.2014.part04.xml.gz
-rw-r--r-- 1 lars users   145212695 Mar  6 22:30 Names.2014.part05.marc8.gz
-rw-r--r-- 1 lars users   145953182 Mar  6 22:30 Names.2014.part05.utf8.gz
-rw-r--r-- 1 lars users   371973849 Mar  6 22:31 Names.2014.part05.xml.gz
-rw-r--r-- 1 lars users   145844340 Mar  6 22:32 Names.2014.part06.marc8.gz
-rw-r--r-- 1 lars users   146731035 Mar  6 22:33 Names.2014.part06.utf8.gz
-rw-r--r-- 1 lars users   379654492 Mar  6 22:33 Names.2014.part06.xml.gz
-rw-r--r-- 1 lars users   139730108 Mar  6 22:34 Names.2014.part07.marc8.gz
-rw-r--r-- 1 lars users   140542064 Mar  6 22:35 Names.2014.part07.utf8.gz
-rw-r--r-- 1 lars users   353551399 Mar  6 22:35 Names.2014.part07.xml.gz
-rw-r--r-- 1 lars users   136203133 Mar  6 22:36 Names.2014.part08.marc8.gz
-rw-r--r-- 1 lars users   136783970 Mar  6 22:37 Names.2014.part08.utf8.gz
-rw-r--r-- 1 lars users   350890835 Mar  6 22:37 Names.2014.part08.xml.gz
-rw-r--r-- 1 lars users   131315036 Mar  6 22:38 Names.2014.part09.marc8.gz
-rw-r--r-- 1 lars users   131805101 Mar  6 22:39 Names.2014.part09.utf8.gz
-rw-r--r-- 1 lars users   327613055 Mar  6 22:39 Names.2014.part09.xml.gz
-rw-r--r-- 1 lars users   134580163 Mar  6 22:40 Names.2014.part10.marc8.gz
-rw-r--r-- 1 lars users   135348785 Mar  6 22:41 Names.2014.part10.utf8.gz
-rw-r--r-- 1 lars users   335096157 Mar  6 22:41 Names.2014.part10.xml.gz
-rw-r--r-- 1 lars users   127972301 Mar  6 22:42 Names.2014.part11.marc8.gz
-rw-r--r-- 1 lars users   129035112 Mar  6 22:43 Names.2014.part11.utf8.gz
-rw-r--r-- 1 lars users   316462193 Mar  6 22:43 Names.2014.part11.xml.gz
-rw-r--r-- 1 lars users   125377432 Mar  6 22:44 Names.2014.part12.marc8.gz
-rw-r--r-- 1 lars users   126177972 Mar  6 22:44 Names.2014.part12.utf8.gz
-rw-r--r-- 1 lars users   311720539 Mar  6 22:45 Names.2014.part12.xml.gz
-rw-r--r-- 1 lars users   119734364 Mar  6 22:46 Names.2014.part13.marc8.gz
-rw-r--r-- 1 lars users   120467491 Mar  6 22:46 Names.2014.part13.utf8.gz
-rw-r--r-- 1 lars users   303515618 Mar  6 22:47 Names.2014.part13.xml.gz
-rw-r--r-- 1 lars users   118218580 Mar  6 22:48 Names.2014.part14.marc8.gz
-rw-r--r-- 1 lars users   118981650 Mar  6 22:48 Names.2014.part14.utf8.gz
-rw-r--r-- 1 lars users   299863852 Mar  6 22:49 Names.2014.part14.xml.gz
-rw-r--r-- 1 lars users   127482102 Mar  6 22:49 Names.2014.part15.marc8.gz
-rw-r--r-- 1 lars users   128212778 Mar  6 22:50 Names.2014.part15.utf8.gz
-rw-r--r-- 1 lars users   316284204 Mar  6 22:50 Names.2014.part15.xml.gz
-rw-r--r-- 1 lars users   120132655 Mar  6 22:51 Names.2014.part16.marc8.gz
-rw-r--r-- 1 lars users   120990890 Mar  6 22:52 Names.2014.part16.utf8.gz
-rw-r--r-- 1 lars users   305355746 Mar  6 22:52 Names.2014.part16.xml.gz
-rw-r--r-- 1 lars users   121497164 Mar  6 22:53 Names.2014.part17.marc8.gz
-rw-r--r-- 1 lars users   122268210 Mar  6 22:53 Names.2014.part17.utf8.gz
-rw-r--r-- 1 lars users   306183075 Mar  6 22:54 Names.2014.part17.xml.gz
-rw-r--r-- 1 lars users   121604332 Mar  6 22:54 Names.2014.part18.marc8.gz
-rw-r--r-- 1 lars users   122514902 Mar  6 22:55 Names.2014.part18.utf8.gz
-rw-r--r-- 1 lars users   304168406 Mar  6 22:56 Names.2014.part18.xml.gz
-rw-r--r-- 1 lars users   118993765 Mar  6 22:56 Names.2014.part19.marc8.gz
-rw-r--r-- 1 lars users   119824991 Mar  6 22:57 Names.2014.part19.utf8.gz
-rw-r--r-- 1 lars users   293192329 Mar  6 22:57 Names.2014.part19.xml.gz
-rw-r--r-- 1 lars users   118402725 Mar  6 22:58 Names.2014.part20.marc8.gz
-rw-r--r-- 1 lars users   118993483 Mar  6 22:58 Names.2014.part20.utf8.gz
-rw-r--r-- 1 lars users   289940225 Mar  6 22:59 Names.2014.part20.xml.gz
-rw-r--r-- 1 lars users   123808537 Mar  6 22:59 Names.2014.part21.marc8.gz
-rw-r--r-- 1 lars users   124213381 Mar  6 23:00 Names.2014.part21.utf8.gz
-rw-r--r-- 1 lars users   312402021 Mar  6 23:01 Names.2014.part21.xml.gz
-rw-r--r-- 1 lars users   109872886 Mar  6 23:01 Names.2014.part22.marc8.gz
-rw-r--r-- 1 lars users   110191969 Mar  6 23:01 Names.2014.part22.utf8.gz
-rw-r--r-- 1 lars users   285322141 Mar  6 23:02 Names.2014.part22.xml.gz
-rw-r--r-- 1 lars users   134687884 Mar  6 23:03 Names.2014.part23.marc8.gz
-rw-r--r-- 1 lars users   135188738 Mar  6 23:03 Names.2014.part23.utf8.gz
-rw-r--r-- 1 lars users   349312546 Mar  6 23:04 Names.2014.part23.xml.gz
-rw-r--r-- 1 lars users   139656833 Mar  6 23:05 Names.2014.part24.marc8.gz
-rw-r--r-- 1 lars users   140429254 Mar  6 23:05 Names.2014.part24.utf8.gz
-rw-r--r-- 1 lars users   343070107 Mar  6 23:06 Names.2014.part24.xml.gz
-rw-r--r-- 1 lars users   130367583 Mar  6 23:07 Names.2014.part25.marc8.gz
-rw-r--r-- 1 lars users   130896729 Mar  6 23:07 Names.2014.part25.utf8.gz
-rw-r--r-- 1 lars users   333671234 Mar  6 23:08 Names.2014.part25.xml.gz
-rw-r--r-- 1 lars users   134496047 Mar  6 23:09 Names.2014.part26.marc8.gz
-rw-r--r-- 1 lars users   135319895 Mar  6 23:09 Names.2014.part26.utf8.gz
-rw-r--r-- 1 lars users   343687721 Mar  6 23:10 Names.2014.part26.xml.gz
-rw-r--r-- 1 lars users   131694295 Mar  6 23:10 Names.2014.part27.marc8.gz
-rw-r--r-- 1 lars users   132405240 Mar  6 23:11 Names.2014.part27.utf8.gz
-rw-r--r-- 1 lars users   337884658 Mar  6 23:12 Names.2014.part27.xml.gz
-rw-r--r-- 1 lars users   131068213 Mar  6 23:13 Names.2014.part28.marc8.gz
-rw-r--r-- 1 lars users   131739682 Mar  6 23:13 Names.2014.part28.utf8.gz
-rw-r--r-- 1 lars users   336182701 Mar  6 23:14 Names.2014.part28.xml.gz
-rw-r--r-- 1 lars users   128283863 Mar  6 23:14 Names.2014.part29.marc8.gz
-rw-r--r-- 1 lars users   128834138 Mar  6 23:15 Names.2014.part29.utf8.gz
-rw-r--r-- 1 lars users   328680794 Mar  6 23:16 Names.2014.part29.xml.gz
-rw-r--r-- 1 lars users   128980689 Mar  6 23:16 Names.2014.part30.marc8.gz
-rw-r--r-- 1 lars users   129466370 Mar  6 23:17 Names.2014.part30.utf8.gz
-rw-r--r-- 1 lars users   329206568 Mar  6 23:17 Names.2014.part30.xml.gz
-rw-r--r-- 1 lars users   129785810 Mar  6 23:18 Names.2014.part31.marc8.gz
-rw-r--r-- 1 lars users   130178917 Mar  6 23:19 Names.2014.part31.utf8.gz
-rw-r--r-- 1 lars users   328737993 Mar  6 23:19 Names.2014.part31.xml.gz
-rw-r--r-- 1 lars users   131094474 Mar  6 23:20 Names.2014.part32.marc8.gz
-rw-r--r-- 1 lars users   131502729 Mar  6 23:20 Names.2014.part32.utf8.gz
-rw-r--r-- 1 lars users   330980471 Mar  6 23:21 Names.2014.part32.xml.gz
-rw-r--r-- 1 lars users   135871719 Mar  6 23:21 Names.2014.part33.marc8.gz
-rw-r--r-- 1 lars users   136263324 Mar  6 23:22 Names.2014.part33.utf8.gz
-rw-r--r-- 1 lars users   341795239 Mar  6 23:23 Names.2014.part33.xml.gz
-rw-r--r-- 1 lars users   159542440 Mar  6 23:24 Names.2014.part34.marc8.gz
-rw-r--r-- 1 lars users   160035758 Mar  6 23:24 Names.2014.part34.utf8.gz
-rw-r--r-- 1 lars users   410421877 Mar  6 23:25 Names.2014.part34.xml.gz
-rw-r--r-- 1 lars users   145741314 Mar  6 23:26 Names.2014.part35.marc8.gz
-rw-r--r-- 1 lars users   146204757 Mar  6 23:26 Names.2014.part35.utf8.gz
-rw-r--r-- 1 lars users   370756365 Mar  6 23:27 Names.2014.part35.xml.gz
-rw-r--r-- 1 lars users   136505530 Mar  6 23:27 Names.2014.part36.marc8.gz
-rw-r--r-- 1 lars users   137112741 Mar  6 23:28 Names.2014.part36.utf8.gz
-rw-r--r-- 1 lars users   348055966 Mar  6 23:28 Names.2014.part36.xml.gz
-rw-r--r-- 1 lars users   119243589 Mar  6 23:29 Names.2014.part37.marc8.gz
-rw-r--r-- 1 lars users   119750038 Mar  6 23:29 Names.2014.part37.utf8.gz
-rw-r--r-- 1 lars users   301444099 Mar  6 23:29 Names.2014.part37.xml.gz
-rw-r--r-- 1 lars users  1574745266 Mar  6 21:38 Serials.2014.combined.marc8.gz
-rw-r--r-- 1 lars users  1577294477 Mar  6 21:45 Serials.2014.combined.utf8.gz
-rw-r--r-- 1 lars users   153000046 Mar  6 21:46 Serials.2014.part01.marc8.gz
-rw-r--r-- 1 lars users   153339863 Mar  6 21:46 Serials.2014.part01.utf8.gz
-rw-r--r-- 1 lars users   161216715 Mar  6 21:48 Serials.2014.part02.marc8.gz
-rw-r--r-- 1 lars users   161394544 Mar  6 21:48 Serials.2014.part02.utf8.gz
-rw-r--r-- 1 lars users   156260609 Mar  6 21:49 Serials.2014.part03.marc8.gz
-rw-r--r-- 1 lars users   156436261 Mar  6 21:50 Serials.2014.part03.utf8.gz
-rw-r--r-- 1 lars users   159641872 Mar  6 21:50 Serials.2014.part04.marc8.gz
-rw-r--r-- 1 lars users   159703811 Mar  6 21:51 Serials.2014.part04.utf8.gz
-rw-r--r-- 1 lars users   161607090 Mar  6 21:51 Serials.2014.part05.marc8.gz
-rw-r--r-- 1 lars users   161700029 Mar  6 21:52 Serials.2014.part05.utf8.gz
-rw-r--r-- 1 lars users   159352935 Mar  6 21:53 Serials.2014.part06.marc8.gz
-rw-r--r-- 1 lars users   160014445 Mar  6 21:54 Serials.2014.part06.utf8.gz
-rw-r--r-- 1 lars users   139575613 Mar  6 21:54 Serials.2014.part07.marc8.gz
-rw-r--r-- 1 lars users   140115302 Mar  6 21:55 Serials.2014.part07.utf8.gz
-rw-r--r-- 1 lars users   127774934 Mar  6 21:56 Serials.2014.part08.marc8.gz
-rw-r--r-- 1 lars users   127861052 Mar  6 21:56 Serials.2014.part08.utf8.gz
-rw-r--r-- 1 lars users   118919822 Mar  6 21:57 Serials.2014.part09.marc8.gz
-rw-r--r-- 1 lars users   119018637 Mar  6 21:57 Serials.2014.part09.utf8.gz
-rw-r--r-- 1 lars users   132217133 Mar  6 21:58 Serials.2014.part10.marc8.gz
-rw-r--r-- 1 lars users   132383827 Mar  6 21:59 Serials.2014.part10.utf8.gz
-rw-r--r-- 1 lars users   105178497 Mar  6 21:59 Serials.2014.part11.marc8.gz
-rw-r--r-- 1 lars users   105326706 Mar  6 22:00 Serials.2014.part11.utf8.gz
-rw-r--r-- 1 lars users   204494374 Mar  6 22:15 Subjects.2014.marc8.gz
-rw-r--r-- 1 lars users    96991643 Mar  6 22:16 Subjects.2014.part01.marc8.gz
-rw-r--r-- 1 lars users    97125788 Mar  6 22:16 Subjects.2014.part01.utf8.gz
-rw-r--r-- 1 lars users   271018292 Mar  6 22:17 Subjects.2014.part01.xml.gz
-rw-r--r-- 1 lars users   107502731 Mar  6 22:17 Subjects.2014.part02.marc8.gz
-rw-r--r-- 1 lars users   107761308 Mar  6 22:18 Subjects.2014.part02.utf8.gz
-rw-r--r-- 1 lars users   262773760 Mar  6 22:18 Subjects.2014.part02.xml.gz
-rw-r--r-- 1 lars users   204887096 Mar  6 22:19 Subjects.2014.utf8.gz
-rw-r--r-- 1 lars users   533792052 Mar  6 22:20 Subjects.2014.xml.gz
-rw-r--r-- 1 lars users   309641400 Mar  6 21:12 Visual.Materials.2014.part01.marc8.gz
-rw-r--r-- 1 lars users   309869230 Mar  6 21:13 Visual.Materials.2014.part01.utf8.gz
-rw-r--r-- 1 lars users   810529940 Mar  6 21:14 Visual.Materials.2014.part01.xml.gz
-rw-r--r-- 1 lars users       41614 May 17 22:59 index.html
-rw-r--r-- 1 lars users       41614 May 17 22:59 index.html?C=D;O=A
-rw-r--r-- 1 lars users       46087 May 17 17:57 index.html?C=D;O=D
-rw-r--r-- 1 lars users       41614 May 17 22:59 index.html?C=M;O=A
-rw-r--r-- 1 lars users       46087 May 17 17:57 index.html?C=M;O=D
-rw-r--r-- 1 lars users       46087 May 17 17:57 index.html?C=N;O=A
-rw-r--r-- 1 lars users       41614 May 17 22:59 index.html?C=N;O=D
-rw-r--r-- 1 lars users       41614 May 17 22:59 index.html?C=S;O=A
-rw-r--r-- 1 lars users       46087 May 17 17:57 index.html?C=S;O=D
-rw-r--r-- 1 lars users   328737993 May 17 17:41 test-Names.2014.part31.xml
-rw-r--r-- 1 lars users           0 Apr  4 19:47 text.txt
ghost commented 7 years ago

We have completely different file sizes. What's the output of file BooksAll.2014.part01.utf8.gz?

ghost commented 7 years ago 
> file BooksAll.2014.part01.utf8.gz
BooksAll.2014.part01.utf8.gz: MARC21 Bibliographic
> sha256sum BooksAll.2014.part01.utf8.gz 
6a3bb078b8a905282419caa2e43f83ab6c93ea08cad6878ca8f2f0f3dd9a2029  BooksAll.2014.part01.utf8.gz
> ipfs add -n BooksAll.2014.part01.utf8.gz 
added QmP6Fnp7pJJ6R8ZXQ77c18HY9oF1nLnhF8mBXWrSuwyGTz BooksAll.2014.part01.utf8.gz
ghost commented 7 years ago

BooksAll.2014.part01.utf8.gz: gzip compressed data, was "BooksAll.part01.utf8", last modified: Wed Dec 9 14:21:29 2015, from Unix

As I suspected, your data is uncompressed - I don't know how.

root@ip-172-30-0-114:/tmp# gzip -d BooksAll.2014.part01.utf8.gz
root@ip-172-30-0-114:/tmp# file BooksAll.2014.part01.utf8
BooksAll.2014.part01.utf8: MARC21 Bibliographic
ghost commented 7 years ago

Mmmh that is weird. Maybe that was wget?

ghost commented 7 years ago

Unknown but it'd be better to host a compressed copy for bandwidth reasons.

ghost commented 7 years ago

Okay I'm making another download and this time it seems correct:

> file loc.gov/cds/downloads/MDSConnect/BooksAll.2014.part01.utf8.gz 
loc.gov/cds/downloads/MDSConnect/BooksAll.2014.part01.utf8.gz: gzip compressed data, was "BooksAll.part01.utf8", from Unix, last modified: Wed Dec  9 15:21:29 2015
ghost commented 7 years ago

Okay, fixed dataset is here: https://ipfs.io/ipfs/QmcvfB6pAqUfTnuAK8zFKVxbdhopnBPveJrDcy1JAA7HX5

ghost commented 7 years ago

@voltagex did you get the same hash?