Source import tracking

[xhero]

• [x] Remove 856 when it is only a link to a Digital Object
• [x] 653000793: contains 772 $w653000793, crashes on import
• [x] 806629866: contains 772 $w806629866, crashes on import
• [x] 806934200: as above, also has a 773 pointing to itself
• [x] 806156846: idem
• [x] 806533416: idem
• [x] 806532745: idem
• [x] 806553621: idem
• [x] 553006246: idem
• [x] 303000793: 240 10$a contains an exceptional quantity of whitespace, record import crashes
• [x] 455041617: empty $0 in 710 tag
• [x] 1000000575: 130 with $m but no $0
• [x] 1000000584: 700 with $4 pbl but no $0
• [x] 1000000594: 130 with $m but no $0
• [x] 1000000603: 700 with $4 but no 40
• [x] 950000002: 773 with $w0
• [x] A number of records have extremely long 240, with data such as I: Singst. - 2, 2, EHr., 2, Bkl., 2 - 4, 3, 3, 0 - Pk. - Hfe. - Str.; II: Singst. - 2, 2, 2, 2 - 2, 0, 0, 0 - Hfe. - Str.; III: Singst. - 2, 2, 2, 2 - 2, 0, 0, 0 - Str.; IV: Singst. - 2, 2, 2, Bkl., 2 - 2, 2, 0, 0 - Hfe. - Str.; V: Singst. - 1, 1, 1, Bkl., 1 - 0, 0, 0, 0 - Str. (ohne Kb.); VI: Singst. - Picc., 2, 2, EHr., 2, Bkl., 2, Kfg. - 4, 2, 3, 1 - Pk., Tamtam, grTr. - 2 Hfe. - Str. (14, 14, 12, 8, 6) These are: 458001613 458001658 458001659 458001660 458001662 458001663 458001664 458001666 458001673 458001674 458001675 458001676 458001677 458001710 458001711 458001732 458001734 458001735 458001767 458001769 458001771 458001773 458001775 458001854 458001855 458001856 458001857 458001865 458001890 458001904 458001915 458001958 458001959 458001960 458001961 458001962 458001963 458001964 458001965 458001966 458001967 458001968 458001974 458001975 458001978 458001983 458001984 458001985 458001987 458002007 458002009 458002010 458002014 458002015 458002016 458002017 458002043 458002044 458002045 458002046
• [x] A number of records have 852 $a but empty $x, and create problems when making the links: 990023495 990023495 990023497 990060619 1000000656 993114943 993109303 993109304 993109305 993109306 993109307 993109308 993109309 993109310 993109311 993109312 993109313 993109314 993109315 993109316 993109317 993109318 993109319 993109320 993109321 993109322 993109323 993109324 993109325 993109326 993109327 993109328 993109329 993109330 993109331 993109333 993109334 993109335 993109336 993109337 993109338 993109339 993109340 993109341 993109342 993109343 993109344 993109345 993109346 993109347 993109348 993109349
• [x] Fix records with 700 and no $0 458001409 458001410 458001411 458001414 458001432 458001433 458001434 458001435 458001456 458001457 458001458 458001466 458001467 458001470 458001477 458001478 458001482 458001485 458001486 458001487 458001488 458001500 458001504 458001505 458001506 458001507 458001508 458001509 458001510 458001511 458001512 458001513 458001514 458001515 458001516 458001517 458001518 458001519 458001520 458001521 458001522 458001528 458001530 458001541 458001542 458001543 458001544 458001545 458001546 458001547 458001548 458001549 458001558 458001559 458001561 458001566 458001576 458001577 458001578 458001579 458001580 458001581 458001582 458001583 458001584 458001585 458001586 458001587 458001588 458001589 458001590 458001596 458001597 458001598 458001599 458001600 458001601 458001602 458001603 458001604 458001605 458001612 458001614 458001615 458001616 458001617 458001643 458001644 458001645 458001646 458001647 458001648 458001649 458001650 458001651 458001652 458001653 458001654 458001655 458001656 458001657 458001661 458001667 458001668 458001669 458001671 458001672 458001678 458001679 458001680 458001681 458001695 458001696 458001697 458001698 458001699 458001700 458001701 458001702 458001703 458001704 458001705 458001706 458001707 458001708 458001709 458001712 458001713 458001714 458001715 458001716 458001717 458001718 458001719 458001722 458001723 458001724 458001729 458001730 458001731 458001733 458001737 458001738 458001739 458001740 458001743 458001744 458001745 458001746 458001750 458001751 458001752 458001753 458001754 458001755 458001756 458001757 458001765 458001768 458001770 458001772 458001774 458001776 458001777 458001778 458001779 458001780 458001781 458001782 458001783 458001784 458001785 458001786 458001787 458001788 458001789 458001790 458001791 458001792 458001793 458001794 458001795 458001796 458001797 458001798 458001799 458001800 458001801 458001802 458001803 458001804 458001805 458001806 458001807 458001808 458001809 458001810 458001811 458001812 458001813 458001814 458001815 458001816 458001817 458001818 458001819 458001820 458001821 458001822 458001823 458001846 458001847 458001848 458001849 458001850 458001851 458001852 458001853 458001858 458001859 458001860 458001861 458001862 458001863 458001864 458001866 458001868 458001871 458001879 458001889 458001892 458001893 458001894 458001895 458001896 458001897 458001898 458001899 458001900 458001901 458001902 458001903 458001907 458001908 458001909 458001910 458001911 458001912 458001913 458001914 458001916 458001917 458001918 458001919 458001949 458001950 458001970 458001976 458001977 458001979 458001980 458001981 458001982 458001986 458001988 458001989 458001990 458001991 458001992 458001993 458001994 458001998 458002001 458002002 458002004 458002006 458002008 458002011 458002018 458002019 458002020 458002023 458002024 458002025 458002026 458002027 458002028 458002029 458002030 458002031 458002032 458002033 458002034 458002035 458002036 458002037 458002042
• [x] These records are created by reference but remain empty - are they the "no publish"? 45006349 212000022 212003605 212003613 212003626 212003647 212005000 212007251 212007457 212007473 225004125 230011000 240009431 240009503 280005559 280005946 280006176 280006297 280006344 280006430 280006511 280006575 280006625 280006803 280007007 280007151 280007246 290003340 300512688 300512691 300512694 300514215 301010561 301011301 301011473 301011744 303000025 303000041 303005087 303005100 303005244 303005255 303005268 303005279 303005318 303005346 303005352 303005384 303005391 303005395 303005445 303005456 303005472 303005609 310005195 350001538 450058457 450059040 450064617 450106283 450117371 453006633 453012090 453012990 453013173 453450119 453450132 453450140 453450171 453450185 453450197 453450999 454600299 454600655 455035943 455040022 455040423 553006246 600502503 600502553 603002092 603002199 603002426 603002488 605201249 605201401 650012374 650012734 650012741 650012870 650013121 650013402 650013447 650013506 650013601 650014575 650014777 653000793 659002382 659002624 659003197 659003258 659100194 659100247 659100293 659500042 659500157 806156846 806532745 806533416 806553621 806629866 806934200 950000001

[xhero]

For the very log 240 $m: if it is needed as-is we can increase the dimension of std_title to a TEXT field

[lpugin]

I would pretty truncate std_title. It does not make sense to have it that long.

[xhero]

Ok for me - maybe we can pretty truncate the single pieces that make it up? Like Masses (v4); 2vl 3vc etc etc etc etc...; other stuff etc... instead of Masses (4v); impressively long list of instruments...

[lpugin]

Yes, this would be better

[HirschSt]

Truncate is ok, these records should been corrected by hand later

[xhero]

It seems @HirschSt comment was deleted when I clicked the ckeckbox, sorry!

[xhero]

@HirschSt Is the new data for June 01 already available? I would like to run an import

[HirschSt]

@xhero I am working on it, but latest fixes from today would be available tomorrow at the earliest (dataset > 20160601)

[xhero]

Could you also drop the offending 700s without $0? they still create problems when loading/saving/reindexing

[HirschSt]

Ok, I will drop 700 and 710 from sources if $0 missing

[xhero]

@HirschSt could you also add the titles with ""no-publishing"? thanks!

[HirschSt]

@xhero "non-publishing" records are included in dataset >= 20160601

[xhero]

There are still records with the DE-588a IDS in 700, which still created problems. I added a hard-coded fix in to_internal, also for 852 $x

[HirschSt]

852$x will be fixed in data and in the export

[xhero]

Data import does not show any errors anymore - most of these seem to be resolved

[lpugin]

Well done. They are a few remaining checkboxes...

[HirschSt]

I am not sure about dropping 856 ... will it be restored in the export, and where are the images hosted? The remaining three sources still are needing more investigation and probably should be fixed in the application

[xhero]

Indexing now is almost perfect! only two records resist:

• 201012025: =710 1#$aMusikverlag N. Simrock Berlin, Leipzig (Berlin$00$4asn
• 455041617: =710 1#$aMusikverlag N. Simrock Berlin, Leipzig (Berlin$00$4fmo

They both have 710 $0 = 0

@xhero SH: will be fixed with the export >=20160608

[xhero]

Oops, two more:

• 1000000575: Tag 240: missing master (expected in $0), tag contents: =240 10$mB
• 1000000594: Tag 240: missing master (expected in $0), tag contents: =240 10$mS; bc

Not bad, out of 1532188 total reindexed items!

[xhero]

Last check: Import signals these tags as missing:

Subfield 031 $2 missing in the marc configuration
Subfield 594 $a missing in the marc configuration
Subfield 240 $n missing in the marc configuration
Subfield 710 $g missing in the marc configuration
Subfield 772 $t missing in the marc configuration

I checked all of them and they are either dropped or are removed in to_internal, so for me it is OK. I just need this to be double-checked so we have no surprises.

[lpugin]

They will all need to be added back in to_external (together with control fields), so keep the list in the appropriate ticket.

[HirschSt]

I was pretty sure I had added 710$g to sources-conf, so I will add it back (with 54f960a). Other missing tags ok with me too.

[HirschSt]

130 comes from moving 240 to 130 because of the collection record type, so there is no $0; 950000002 needs updating by hand

[HirschSt]

950000002 is corrected in dataset >= 20160609