bonnland
commented
1 year ago Here is an example of a preprocessing run with progress indicator.
The progress lines are formatted using fixed-width fields for strings such as the filename.
(1 of 112): TR_B3 2021_ACM.xlsx : found 291 rows. 01/12/2022 16:56:37
(2 of 112): TR_B3 2021_ACS.xlsx : found 816 rows. 01/12/2022 16:56:37
(3 of 112): TR_B3 2021_AIAA.xlsx : found 3372 rows. 01/12/2022 16:56:38
(4 of 112): TR_B3 2021_ASCE.xlsx : found 1680 rows. 01/12/2022 16:56:39
(5 of 112): TR_B3 2021_ASME.xlsx : found 39 rows. 01/12/2022 16:56:39
(6 of 112): TR_B3 2021_Accessible Archives.xlsx: found 60 rows. 01/12/2022 16:56:39
(7 of 112): TR_B3 2021_American Mathematical Society.xlsx: found 258 rows. 01/12/2022 16:56:39
(8 of 112): TR_B3 2021_American Psychiatric Association.xlsx: found 60 rows. 01/12/2022 16:56:39
(9 of 112): TR_B3 2021_Bloomsbury.xlsx : found 2376 rows. 01/12/2022 16:56:40
(10 of 112): TR_B3 2021_Brepols.xlsx : found 72 rows. 01/12/2022 16:56:40
(11 of 112): TR_B3 2021_Brill.xlsx : found 2608 rows. 01/12/2022 16:56:41
(12 of 112): TR_B3 2021_Cambridge.xlsx : found 8613 rows. 01/12/2022 16:56:45
(13 of 112): TR_B3 2021_De Gruyter.xlsx : found 1146 rows. 01/12/2022 16:56:46
(14 of 112): TR_B3 2021_Duke.xlsx : found 5379 rows. 01/12/2022 16:56:48
(15 of 112): TR_B3 2021_EBSCO.xlsx : found 68163 rows. 01/12/2022 16:57:18
(16 of 112): TR_B3 2021_Emerald.xlsx : found 687 rows. 01/12/2022 16:57:18
(17 of 112): TR_B3 2021_GeoScienceWorld.xlsx: found 1092 rows. 01/12/2022 16:57:19
(18 of 112): TR_B3 2021_ICE.xlsx : found 69 rows. 01/12/2022 16:57:19
(19 of 112): TR_B3 2021_IEEE.xlsx : found 1416 rows. 01/12/2022 16:57:19
(20 of 112): TR_B3 2021_IOP.xlsx : found 942 rows. 01/12/2022 16:57:20
(21 of 112): TR_B3 2021_Ingenta.xlsx : found 6 rows. 01/12/2022 16:57:20
(22 of 112): TR_B3 2021_JSTOR.xlsx : found 14226 rows. 01/12/2022 16:57:28
(23 of 112): TR_B3 2021_Liebert.xlsx : found 3 rows. 01/12/2022 16:57:28
(24 of 112): TR_B3 2021_Liverpool University Press.xlsx: found 33 rows. 01/12/2022 16:57:28
(25 of 112): TR_B3 2021_MIT Press.xlsx : found 231 rows. 01/12/2022 16:57:28
(26 of 112): TR_B3 2021_OECD.xlsx : found 1380 rows. 01/12/2022 16:57:29
(27 of 112): TR_B3 2021_Ovid.xlsx : found 6 rows. 01/12/2022 16:57:29
(28 of 112): TR_B3 2021_Oxford University Press.xlsx: found 11376 rows. 01/12/2022 16:57:37
(29 of 112): TR_B3 2021_ProQuest.xlsx : found 6168 rows. 01/12/2022 16:57:39
(30 of 112): TR_B3 2021_Project Euclid.xlsx: found 162 rows. 01/12/2022 16:57:39
(31 of 112): TR_B3 2021_Project Muse.xlsx : found 2127 rows. 01/12/2022 16:57:41
(32 of 112): TR_B3 2021_Psychiatry Online.xlsx: found 60 rows. 01/12/2022 16:57:41
(33 of 112): TR_B3 2021_RSC.xlsx : found 306 rows. 01/12/2022 16:57:41
(34 of 112): TR_B3 2021_SIAM.xlsx : found 1506 rows. 01/12/2022 16:57:41
(35 of 112): TR_B3 2021_SPIE.xlsx : found 612 rows. 01/12/2022 16:57:42
(36 of 112): TR_B3 2021_Sage.xlsx : found 6693 rows. 01/12/2022 16:57:44
(37 of 112): TR_B3 2021_ScienceDirect.xlsx : found 49851 rows. 01/12/2022 16:58:02
(38 of 112): TR_B3 2021_Springer.xlsx : found 240276 rows. 01/12/2022 16:59:32
(39 of 112): TR_B3 2021_Thieme.xlsx : found 69 rows. 01/12/2022 16:59:32
(40 of 112): TR_B3 2021_Wiley.xlsx : found 7833 rows. 01/12/2022 16:59:34
(41 of 112): TR_B3 2021_WorldBank.xlsx : found 798 rows. 01/12/2022 16:59:35
(42 of 112): TR_J3 2021_ACM.xlsx : found 1006 rows. 01/12/2022 16:59:35
(43 of 112): TR_J3 2021_ACS.xlsx : found 560 rows. 01/12/2022 16:59:35
(44 of 112): TR_J3 2021_AIAA.xlsx : found 90 rows. 01/12/2022 16:59:35
(45 of 112): TR_J3 2021_AK.xlsx : found 90 rows. 01/12/2022 16:59:35
(46 of 112): TR_J3 2021_ASCE.xlsx : found 214 rows. 01/12/2022 16:59:35
(47 of 112): TR_J3 2021_ASME.xlsx : found 1006 rows. 01/12/2022 16:59:36
(48 of 112): TR_J3 2021_ASTM.xlsx : found 16 rows. 01/12/2022 16:59:36
(49 of 112): TR_J3 2021_Accessible Archives.xlsx: found 200 rows. 01/12/2022 16:59:36
(50 of 112): TR_J3 2021_Allen Press.xlsx : found 266 rows. 01/12/2022 16:59:36
(51 of 112): TR_J3 2021_American Economic Association.xlsx: found 36 rows. 01/12/2022 16:59:36
(52 of 112): TR_J3 2021_American Mathematical Society.xlsx: found 46 rows. 01/12/2022 16:59:36
(53 of 112): TR_J3 2021_American Meteorological Society.xlsx: found 92 rows. 01/12/2022 16:59:36
(54 of 112): TR_J3 2021_American Physical Society.xlsx: found 80 rows. 01/12/2022 16:59:36
(55 of 112): TR_J3 2021_American Physiological Society.xlsx: found 100 rows. 01/12/2022 16:59:36
(56 of 112): TR_J3 2021_American Phytopathological Society.xlsx: found 36 rows. 01/12/2022 16:59:36
(57 of 112): TR_J3 2021_American Psychiatric Association.xlsx: found 36 rows. 01/12/2022 16:59:36
(58 of 112): TR_J3 2021_American Society for Microbiology.xlsx: found 108 rows. 01/12/2022 16:59:37
(59 of 112): TR_J3 2021_Annual Reviews.xlsx: found 276 rows. 01/12/2022 16:59:37
(60 of 112): TR_J3 2021_BMJ.xlsx : found 280 rows. 01/12/2022 16:59:37
(61 of 112): TR_J3 2021_Berghahn Journals.xlsx: found 180 rows. 01/12/2022 16:59:37
(62 of 112): TR_J3 2021_BioOne.xlsx : found 696 rows. 01/12/2022 16:59:37
(63 of 112): TR_J3 2021_Brepols.xlsx : found 84 rows. 01/12/2022 16:59:37
(64 of 112): TR_J3 2021_Brill.xlsx : found 606 rows. 01/12/2022 16:59:37
(65 of 112): TR_J3 2021_CAIRN int.xlsx : found 90 rows. 01/12/2022 16:59:37
(66 of 112): TR_J3 2021_CAIRN.xlsx : found 634 rows. 01/12/2022 16:59:38
(67 of 112): TR_J3 2021_CSIRO.xlsx : found 164 rows. 01/12/2022 16:59:38
(68 of 112): TR_J3 2021_Cambridge.xlsx : found 2800 rows. 01/12/2022 16:59:39
(69 of 112): TR_J3 2021_De Gruyter.xlsx : found 1028 rows. 01/12/2022 16:59:39
(70 of 112): TR_J3 2021_Duke.xlsx : found 248 rows. 01/12/2022 16:59:39
(71 of 112): TR_J3 2021_EBSCO.xlsx : found 51718 rows. 01/12/2022 17:00:02
(72 of 112): TR_J3 2021_EDP Sciences.xlsx : found 340 rows. 01/12/2022 17:00:03
(73 of 112): TR_J3 2021_Edinburgh University Press.xlsx: found 134 rows. 01/12/2022 17:00:03
(74 of 112): TR_J3 2021_Emerald.xlsx : found 1504 rows. 01/12/2022 17:00:03
(75 of 112): TR_J3 2021_GeoScienceWorld.xlsx: found 260 rows. 01/12/2022 17:00:03
(76 of 112): TR_J3 2021_Guilford.xlsx : found 44 rows. 01/12/2022 17:00:03
(77 of 112): TR_J3 2021_HighWire.xlsx : found 160 rows. 01/12/2022 17:00:03
(78 of 112): TR_J3 2021_ICE.xlsx : found 154 rows. 01/12/2022 17:00:03
(79 of 112): TR_J3 2021_IEEE.xlsx : found 1898 rows. 01/12/2022 17:00:05
(80 of 112): TR_J3 2021_INFORMS.xlsx : found 98 rows. 01/12/2022 17:00:05
(81 of 112): TR_J3 2021_IOP.xlsx : found 900 rows. 01/12/2022 17:00:06
(82 of 112): TR_J3 2021_Ingenta.xlsx : found 1202 rows. 01/12/2022 17:00:06
(83 of 112): TR_J3 2021_JSTAGE.xlsx : found 54 rows. 01/12/2022 17:00:06
(84 of 112): TR_J3 2021_JSTOR.xlsx : found 13966 rows. 01/12/2022 17:00:11
(85 of 112): TR_J3 2021_Liebert.xlsx : found 602 rows. 01/12/2022 17:00:11
(86 of 112): TR_J3 2021_Liverpool University Press.xlsx: found 12 rows. 01/12/2022 17:00:11
(87 of 112): TR_J3 2021_MIT Press.xlsx : found 216 rows. 01/12/2022 17:00:12
(88 of 112): TR_J3 2021_Microbiology Society.xlsx: found 40 rows. 01/12/2022 17:00:12
(89 of 112): TR_J3 2021_Morgan & Claypool.xlsx: found 154 rows. 01/12/2022 17:00:12
(90 of 112): TR_J3 2021_NRC.xlsx : found 196 rows. 01/12/2022 17:00:12
(91 of 112): TR_J3 2021_OECD .xlsx : found 162 rows. 01/12/2022 17:00:12
(92 of 112): TR_J3 2021_Ovid.xlsx : found 2768 rows. 01/12/2022 17:00:13
(93 of 112): TR_J3 2021_Oxford Academic.xlsx: found 2864 rows. 01/12/2022 17:00:14
(94 of 112): TR_J3 2021_ProQuest.xlsx : found 45984 rows. 01/12/2022 17:00:29
(95 of 112): TR_J3 2021_Project Euclid.xlsx: found 608 rows. 01/12/2022 17:00:30
(96 of 112): TR_J3 2021_Project Muse.xlsx : found 2808 rows. 01/12/2022 17:00:31
(97 of 112): TR_J3 2021_Psychiatry Online.xlsx: found 36 rows. 01/12/2022 17:00:31
(98 of 112): TR_J3 2021_RSC.xlsx : found 420 rows. 01/12/2022 17:00:31
(99 of 112): TR_J3 2021_SIAM.xlsx : found 120 rows. 01/12/2022 17:00:31
(100 of 112): TR_J3 2021_SPIE.xlsx : found 88 rows. 01/12/2022 17:00:31
(101 of 112): TR_J3 2021_Sage Journals.xlsx : found 6558 rows. 01/12/2022 17:00:35
(102 of 112): TR_J3 2021_ScienceDirect.xlsx : found 19610 rows. 01/12/2022 17:00:41
(103 of 112): TR_J3 2021_Scitation.xlsx : found 98 rows. 01/12/2022 17:00:41
(104 of 112): TR_J3 2021_Society for Endocrinology (Bioscientifica).xlsx: found 48 rows. 01/12/2022 17:00:41
(105 of 112): TR_J3 2021_Springer.xlsx : found 21848 rows. 01/12/2022 17:00:49
(106 of 112): TR_J3 2021_T&F.xlsx : found 15010 rows. 01/12/2022 17:00:54
(107 of 112): TR_J3 2021_Thieme.xlsx : found 272 rows. 01/12/2022 17:00:54
(108 of 112): TR_J3 2021_University of California Press.xlsx: found 158 rows. 01/12/2022 17:00:54
(109 of 112): TR_J3 2021_University of Chicago Press.xlsx: found 478 rows. 01/12/2022 17:00:55
(110 of 112): TR_J3 2021_University of Toronto Press.xlsx: found 106 rows. 01/12/2022 17:00:55
(111 of 112): TR_J3 2021_Wiley.xlsx : found 12120 rows. 01/12/2022 17:00:59
(112 of 112): TR_J3 2021_WorldBank.xlsx : found 12 rows. 01/12/2022 17:00:59
Potentially dozens to 100 or more files may need to be pre processed. Indication of progress would help set expectations.
Estimate: 2
Acceptance Criteria:
Indicate progress of pre-processing of bulk file sets. Consider: Indicate "how much work" a file may be based on row count, and how many more files are to be processed. Include timestamps or elapsed time. For each file processed, indicate number of rows for file, file number completed out of total, etc.