search

Rdatatable / data.table

R's data.table package extends data.frame:
http://r-datatable.com
Mozilla Public License 2.0
3.6k stars 982 forks source link

Column number 8 (select[1]) is too large for this table, which only has 3 columns. #5071

Open lime-n opened 3 years ago

lime-n commented 3 years ago

I am trying to read very large .csv files into R that are approximately 10GB each (total 70).

I have a found a function that works well for my situation using fread however, I am starting to get the following issue:

Column number 8 (select[1]) is too large for this table, which only has 3 columns.

Here is the code that I am using:

result <- rbindlist(lapply(fil[-1], function(x) setNames(
  fread(x, select = col_num, header = TRUE), cols)), fill = TRUE) %>% filter(year >= 2019) %>% drop_na()

Now, it seems to work with all files from 1-4, however, the fifth file I am getting the following error above. I have tried to implement this individual for the 5th file, and still get the error:

x <- setNames(fread(fil[2], select = col_num, header = TRUE), cols)

To make sure that I have the right column numbers, and size I have called the file individually and it checks.

Session:

R version 4.1.0 (2021-05-18)
Platform: x86_64-apple-darwin17.0 (64-bit)
Running under: macOS Big Sur 11.4

Matrix products: default
LAPACK: /Library/Frameworks/R.framework/Versions/4.1/Resources/lib/libRlapack.dylib

locale:
[1] en_GB.UTF-8/en_GB.UTF-8/en_GB.UTF-8/C/en_GB.UTF-8/en_GB.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] data.table_1.14.0 tidyr_1.1.3       vroom_1.5.1       purrr_0.3.4       dplyr_1.0.7      

loaded via a namespace (and not attached):
 [1] rstudioapi_0.13  magrittr_2.0.1   tidyselect_1.1.1 bit_4.0.4        R6_2.5.0        
 [6] rlang_0.4.11     fansi_0.5.0      tools_4.1.0      utf8_1.2.1       DBI_1.1.1       
[11] ellipsis_0.3.2   bit64_4.0.5      assertthat_0.2.1 tibble_3.1.2     lifecycle_1.0.0 
[16] crayon_1.4.1     tzdb_0.1.1       vctrs_0.3.8      glue_1.4.2       compiler_4.1.0  
[21] pillar_1.6.1     generics_0.1.0   pkgconfig_2.0.3
MichaelChirico commented 3 years ago

it seems some files have a different structure than you anticipated. try running with verbose=TRUE, and paste in the output of that here if there's still an issue

lime-n commented 3 years ago

Here is the output and the error still continues.

However, if I just read the file as:

> fread(fil[2])

I get this error instead:

Error in fread(fil[2]) : R character strings are limited to 2^31-1 bytes

Here is the verbose output:

This installation of data.table has not been compiled with OpenMP support.
  omp_get_num_procs()            1
  R_DATATABLE_NUM_PROCS_PERCENT  unset (default 50)
  R_DATATABLE_NUM_THREADS        unset
  R_DATATABLE_THROTTLE           unset (default 1024)
  omp_get_thread_limit()         1
  omp_get_max_threads()          1
  OMP_THREAD_LIMIT               unset
  OMP_NUM_THREADS                unset
  RestoreAfterFork               true
  data.table is using 1 threads with throttle==1024. See ?setDTthreads.
Input contains no \n. Taking this to be a filename to open
[01] Check arguments
  Using 1 threads (omp_get_max_threads()=1, nth=1)
  NAstrings = [<<NA>>]
  None of the NAstrings look like numbers.
  show progress = 1
  0/1 column will be read as integer
[02] Opening the file
  Opening file species_all.df//xbv.csv
  File opened, size = 9.77GB (10485760000 bytes).
  Memory mapped ok
[03] Detect and skip BOM
[04] Arrange mmap to be \0 terminated
  \n has been found in the input and different lines can end with different line endings (e.g. mixed \n and \r\n in one file). This is common and ideal.
  File ends abruptly with 'b'. Copying file in RAM. This file is very unusual: it ends abruptly without a final newline, and also its size is a multiple of 4096 bytes. Please properly end the last row with a newline using for example 'echo >> file' to avoid this  copy.
Avoidable 7.562 seconds. This file is very unusual: it ends abruptly without a final newline, and also its size is a multiple of 4096 bytes. Please properly end the last row with a newline using for example 'echo >> file' to avoid this  time to copy.
  File copy in RAM took 7.562 seconds.
[05] Skipping initial rows if needed
  Positioned on line 1 starting: <<9c4a777e8b   453931765   Animalia    >>
[06] Detect separator, quoting rule, and ncolumns
  Detecting sep automatically ...
  sep=','  with 100 lines of 3 fields using quote rule 0
  Detected 3 columns on line 1. This line is either column names or first data row. Line starts as: <<9c4a777e8b    453931765   Animalia    >>
  Quote rule picked = 0
  fill=false and the most number of columns found is 3
[07] Detect column types, good nrow estimate and whether first row is column names
  'header' changed by user from 'auto' to true
  Number of sampling jump points = 100 because (10485760000 bytes from row 1 to eof) / (2 * 51069 jump0size) == 102662
  Type codes (jump 000)    : CCC  Quote rule 0
  A line with too-few fields (2/3) was found on line 1 of sample jump 1. Most likely this jump landed awkwardly so type bumps here will be skipped.
  A line with too-few fields (2/3) was found on line 3 of sample jump 2. Most likely this jump landed awkwardly so type bumps here will be skipped.
  A line with too-few fields (2/3) was found on line 1 of sample jump 3. Most likely this jump landed awkwardly so type bumps here will be skipped.
  A line with too-few fields (2/3) was found on line 1 of sample jump 4. Most likely this jump landed awkwardly so type bumps here will be skipped.
  A line with too-few fields (2/3) was found on line 1 of sample jump 5. Most likely this jump landed awkwardly so type bumps here will be skipped.
  A line with too-few fields (2/3) was found on line 2 of sample jump 6. Most likely this jump landed awkwardly so type bumps here will be skipped.
  A line with too-few fields (2/3) was found on line 1 of sample jump 7. Most likely this jump landed awkwardly so type bumps here will be skipped.
  A line with too-few fields (2/3) was found on line 1 of sample jump 8. Most likely this jump landed awkwardly so type bumps here will be skipped.
  A line with too-few fields (2/3) was found on line 1 of sample jump 9. Most likely this jump landed awkwardly so type bumps here will be skipped.
  A line with too-few fields (2/3) was found on line 1 of sample jump 10. Most likely this jump landed awkwardly so type bumps here will be skipped.
  A line with too-many fields (3/3) was found on line 2 of sample jump 11. Most likely this jump landed awkwardly so type bumps here will be skipped.
  A line with too-many fields (3/3) was found on line 49 of sample jump 13. Most likely this jump landed awkwardly so type bumps here will be skipped.
  A line with too-few fields (2/3) was found on line 1 of sample jump 15. Most likely this jump landed awkwardly so type bumps here will be skipped.
  A line with too-few fields (2/3) was found on line 1 of sample jump 16. Most likely this jump landed awkwardly so type bumps here will be skipped.
  A line with too-few fields (2/3) was found on line 2 of sample jump 17. Most likely this jump landed awkwardly so type bumps here will be skipped.
  A line with too-few fields (2/3) was found on line 1 of sample jump 18. Most likely this jump landed awkwardly so type bumps here will be skipped.
  A line with too-few fields (2/3) was found on line 11 of sample jump 19. Most likely this jump landed awkwardly so type bumps here will be skipped.
  A line with too-many fields (3/3) was found on line 1 of sample jump 20. Most likely this jump landed awkwardly so type bumps here will be skipped.
  A line with too-many fields (3/3) was found on line 2 of sample jump 21. Most likely this jump landed awkwardly so type bumps here will be skipped.
  A line with too-many fields (3/3) was found on line 1 of sample jump 22. Most likely this jump landed awkwardly so type bumps here will be skipped.
  A line with too-few fields (2/3) was found on line 1 of sample jump 23. Most likely this jump landed awkwardly so type bumps here will be skipped.
  A line with too-few fields (2/3) was found on line 1 of sample jump 24. Most likely this jump landed awkwardly so type bumps here will be skipped.
  A line with too-many fields (3/3) was found on line 2 of sample jump 25. Most likely this jump landed awkwardly so type bumps here will be skipped.
  A line with too-many fields (3/3) was found on line 3 of sample jump 26. Most likely this jump landed awkwardly so type bumps here will be skipped.
  A line with too-few fields (2/3) was found on line 3 of sample jump 27. Most likely this jump landed awkwardly so type bumps here will be skipped.
  A line with too-many fields (3/3) was found on line 6 of sample jump 28. Most likely this jump landed awkwardly so type bumps here will be skipped.
  A line with too-few fields (2/3) was found on line 2 of sample jump 29. Most likely this jump landed awkwardly so type bumps here will be skipped.
  A line with too-many fields (3/3) was found on line 11 of sample jump 30. Most likely this jump landed awkwardly so type bumps here will be skipped.
  A line with too-many fields (3/3) was found on line 2 of sample jump 31. Most likely this jump landed awkwardly so type bumps here will be skipped.
  A line with too-few fields (2/3) was found on line 1 of sample jump 32. Most likely this jump landed awkwardly so type bumps here will be skipped.
  A line with too-few fields (1/3) was found on line 1 of sample jump 33. Most likely this jump landed awkwardly so type bumps here will be skipped.
  A line with too-few fields (2/3) was found on line 2 of sample jump 34. Most likely this jump landed awkwardly so type bumps here will be skipped.
  A line with too-few fields (2/3) was found on line 2 of sample jump 39. Most likely this jump landed awkwardly so type bumps here will be skipped.
  A line with too-few fields (2/3) was found on line 1 of sample jump 40. Most likely this jump landed awkwardly so type bumps here will be skipped.
  A line with too-few fields (2/3) was found on line 2 of sample jump 41. Most likely this jump landed awkwardly so type bumps here will be skipped.
  A line with too-few fields (2/3) was found on line 1 of sample jump 42. Most likely this jump landed awkwardly so type bumps here will be skipped.
  A line with too-many fields (3/3) was found on line 2 of sample jump 43. Most likely this jump landed awkwardly so type bumps here will be skipped.
  A line with too-few fields (2/3) was found on line 1 of sample jump 44. Most likely this jump landed awkwardly so type bumps here will be skipped.
  A line with too-many fields (3/3) was found on line 1 of sample jump 45. Most likely this jump landed awkwardly so type bumps here will be skipped.
  A line with too-many fields (3/3) was found on line 1 of sample jump 46. Most likely this jump landed awkwardly so type bumps here will be skipped.
  A line with too-few fields (2/3) was found on line 1 of sample jump 47. Most likely this jump landed awkwardly so type bumps here will be skipped.
  A line with too-many fields (3/3) was found on line 61 of sample jump 48. Most likely this jump landed awkwardly so type bumps here will be skipped.
  A line with too-few fields (2/3) was found on line 1 of sample jump 49. Most likely this jump landed awkwardly so type bumps here will be skipped.
  A line with too-few fields (2/3) was found on line 1 of sample jump 50. Most likely this jump landed awkwardly so type bumps here will be skipped.
  A line with too-many fields (3/3) was found on line 51 of sample jump 51. Most likely this jump landed awkwardly so type bumps here will be skipped.
  A line with too-few fields (2/3) was found on line 31 of sample jump 52. Most likely this jump landed awkwardly so type bumps here will be skipped.
  A line with too-many fields (3/3) was found on line 1 of sample jump 53. Most likely this jump landed awkwardly so type bumps here will be skipped.
  A line with too-few fields (2/3) was found on line 1 of sample jump 54. Most likely this jump landed awkwardly so type bumps here will be skipped.
  A line with too-few fields (2/3) was found on line 1 of sample jump 55. Most likely this jump landed awkwardly so type bumps here will be skipped.
  A line with too-few fields (2/3) was found on line 1 of sample jump 56. Most likely this jump landed awkwardly so type bumps here will be skipped.
  A line with too-few fields (2/3) was found on line 1 of sample jump 57. Most likely this jump landed awkwardly so type bumps here will be skipped.
  A line with too-few fields (2/3) was found on line 1 of sample jump 60. Most likely this jump landed awkwardly so type bumps here will be skipped.
  A line with too-few fields (2/3) was found on line 1 of sample jump 61. Most likely this jump landed awkwardly so type bumps here will be skipped.
  A line with too-few fields (2/3) was found on line 1 of sample jump 62. Most likely this jump landed awkwardly so type bumps here will be skipped.
  A line with too-few fields (2/3) was found on line 1 of sample jump 63. Most likely this jump landed awkwardly so type bumps here will be skipped.
  A line with too-many fields (3/3) was found on line 2 of sample jump 64. Most likely this jump landed awkwardly so type bumps here will be skipped.
  A line with too-few fields (2/3) was found on line 1 of sample jump 65. Most likely this jump landed awkwardly so type bumps here will be skipped.
  A line with too-few fields (2/3) was found on line 1 of sample jump 66. Most likely this jump landed awkwardly so type bumps here will be skipped.
  A line with too-few fields (2/3) was found on line 1 of sample jump 67. Most likely this jump landed awkwardly so type bumps here will be skipped.
  A line with too-few fields (2/3) was found on line 2 of sample jump 68. Most likely this jump landed awkwardly so type bumps here will be skipped.
  A line with too-many fields (3/3) was found on line 1 of sample jump 69. Most likely this jump landed awkwardly so type bumps here will be skipped.
  A line with too-many fields (3/3) was found on line 2 of sample jump 70. Most likely this jump landed awkwardly so type bumps here will be skipped.
  A line with too-few fields (2/3) was found on line 1 of sample jump 71. Most likely this jump landed awkwardly so type bumps here will be skipped.
  A line with too-few fields (2/3) was found on line 1 of sample jump 72. Most likely this jump landed awkwardly so type bumps here will be skipped.
  A line with too-few fields (2/3) was found on line 1 of sample jump 73. Most likely this jump landed awkwardly so type bumps here will be skipped.
  A line with too-few fields (2/3) was found on line 2 of sample jump 74. Most likely this jump landed awkwardly so type bumps here will be skipped.
  A line with too-few fields (2/3) was found on line 1 of sample jump 75. Most likely this jump landed awkwardly so type bumps here will be skipped.
  A line with too-few fields (2/3) was found on line 23 of sample jump 76. Most likely this jump landed awkwardly so type bumps here will be skipped.
  A line with too-few fields (2/3) was found on line 1 of sample jump 77. Most likely this jump landed awkwardly so type bumps here will be skipped.
  A line with too-few fields (2/3) was found on line 61 of sample jump 78. Most likely this jump landed awkwardly so type bumps here will be skipped.
  A line with too-few fields (1/3) was found on line 8 of sample jump 79. Most likely this jump landed awkwardly so type bumps here will be skipped.
  A line with too-few fields (2/3) was found on line 1 of sample jump 80. Most likely this jump landed awkwardly so type bumps here will be skipped.
  A line with too-few fields (2/3) was found on line 1 of sample jump 81. Most likely this jump landed awkwardly so type bumps here will be skipped.
  A line with too-few fields (2/3) was found on line 1 of sample jump 82. Most likely this jump landed awkwardly so type bumps here will be skipped.
  A line with too-few fields (2/3) was found on line 1 of sample jump 83. Most likely this jump landed awkwardly so type bumps here will be skipped.
  A line with too-few fields (2/3) was found on line 1 of sample jump 84. Most likely this jump landed awkwardly so type bumps here will be skipped.
  A line with too-many fields (3/3) was found on line 4 of sample jump 85. Most likely this jump landed awkwardly so type bumps here will be skipped.
  A line with too-few fields (2/3) was found on line 1 of sample jump 86. Most likely this jump landed awkwardly so type bumps here will be skipped.
  A line with too-few fields (2/3) was found on line 1 of sample jump 87. Most likely this jump landed awkwardly so type bumps here will be skipped.
  A line with too-few fields (2/3) was found on line 1 of sample jump 88. Most likely this jump landed awkwardly so type bumps here will be skipped.
  A line with too-many fields (3/3) was found on line 24 of sample jump 89. Most likely this jump landed awkwardly so type bumps here will be skipped.
  A line with too-many fields (3/3) was found on line 82 of sample jump 90. Most likely this jump landed awkwardly so type bumps here will be skipped.
  A line with too-many fields (3/3) was found on line 1 of sample jump 91. Most likely this jump landed awkwardly so type bumps here will be skipped.
  A line with too-many fields (3/3) was found on line 1 of sample jump 92. Most likely this jump landed awkwardly so type bumps here will be skipped.
  A line with too-many fields (3/3) was found on line 1 of sample jump 93. Most likely this jump landed awkwardly so type bumps here will be skipped.
  A line with too-many fields (3/3) was found on line 61 of sample jump 94. Most likely this jump landed awkwardly so type bumps here will be skipped.
  A line with too-many fields (3/3) was found on line 5 of sample jump 95. Most likely this jump landed awkwardly so type bumps here will be skipped.
  A line with too-many fields (3/3) was found on line 2 of sample jump 96. Most likely this jump landed awkwardly so type bumps here will be skipped.
  A line with too-many fields (3/3) was found on line 1 of sample jump 97. Most likely this jump landed awkwardly so type bumps here will be skipped.
  A line with too-many fields (3/3) was found on line 3 of sample jump 98. Most likely this jump landed awkwardly so type bumps here will be skipped.
  A line with too-few fields (2/3) was found on line 1 of sample jump 99. Most likely this jump landed awkwardly so type bumps here will be skipped.
  A line with too-few fields (2/3) was found on line 1 of sample jump 100. Most likely this jump landed awkwardly so type bumps here will be skipped.
  Type codes (jump 100)    : CCC  Quote rule 0
  =====
  Sampled 1397 rows (handled \n inside quoted fields) at 101 jump points
  Bytes from first data row on line 2 to the end of last row: 10485759533
  Line length: mean=561.03 sd=88.14 min=410 max=732
  Estimated number of rows: 10485759533 / 561.03 = 18690359
  Initial alloc = 25575023 rows (18690359 + 36%) using bytes/max(mean-2*sd,min) clamped between [1.1*estn, 2.0*estn]
  =====
[08] Assign column names
[09] Apply user overrides on column types
MichaelChirico commented 3 years ago

you can see here:

  Detected 3 columns on line 1. This line is either column names or first data row. Line starts as: <<9c4a777e8b    453931765   Animalia    >>

plus header=TRUE so the first line is assumed correct. that file only has 3 fields.

lime-n commented 3 years ago

This is wrong - These files have been split from a larger .csv file and they should all have 50 columns. Just to double check, I read this file with vroom using delim="\t" and it gives me 17 columns. I opt for fread because it reads all columns (most of the time) compared to others.

This does not happen on all the files, probably 5/70 however I think its because the column names is the name of the first row and only the first file has the original column names hence the code I am using renames them.

I had the same issue with a previous file, and calling:

fread(fil[1])

instead of using rbindList and the rest worked, however it creates an error as mentioned previously when passed for this file.

MichaelChirico commented 3 years ago

could you share readChar(f, 500) for the problematic file?

lime-n commented 3 years ago

Here is the data as requested and perhaps it may be a problem with empty rows?:

"\t6066527\tHUMAN_OBSERVATION\tFIAO\tSABAP2\turn:fiao:sabap2:fullprot:rid3033716\t\tMr J.A. Gouws\t\tCC_BY_4_0\t\tMr J.A. Gouws\t\t\t2021-06-01T09:33:06.461Z\t\tCOORDINATE_ROUNDED;GEODETIC_DATUM_ASSUMED_WGS84\n2341768403\t906e6978-e292-4a8b-9c39-adf6bb0f3323\turn:fiao:sabap2:fullprot:rid3035093\tAnimalia\tChordata\tAves\tPasseriformes\tLeiothrichidae\tTurdoides\tTurdoides jardineii\t\tSPECIES\tTurdoides jardineii (A.Smith, 1836)\t\t\tZA\t\tNorth West\tPRESENT\t\tdd862d06-e6e9-4ab9-bc86-c875cc148c1f\t-24.707917\t26.374583\t\t\t\t\t\t\t2011-06-21T00:00:00\t21\t6\t2011\t2493313\t2493313\tHUMAN_OBSERVATION\tFIAO\tSABAP2\turn:fiao:sabap2:fullprot:rid3035093\t\tMr R Hawkins\t\tCC_BY_4_0\t\tMr R Hawkins\t\t\t2021-06-01T09:33:05.201Z\t\tCOORDINATE_ROUNDED;GEODETIC_DATUM_ASSUMED_WGS84\n2341771360\t906e6978-e292-4a8b-9c39-adf6bb0f3323\turn:fiao:sabap2:fullprot:rid3036"
MichaelChirico commented 3 years ago

Hmm that's odd. It looks like maybe skip is doing something funny.

What's gregexpr("9c4a777e8b", readChar(f, file.size(f)))[[1]]?

Could you also try running with setting skip=0?

lime-n commented 3 years ago

skip=0 does the trick! - Spoke too soon ... Seems to not work for other files

Although, I get the following error when running this code:

gregexpr("9c4a777e8b", readChar(f, file.size(f)))[[1]]

Error in readChar(fil[2], file.size(fil[2])) : invalid 'nchars' argument In addition: Warning message: In readChar(fil[2], file.size(fil[2])) : NAs introduced by coercion to integer range

MichaelChirico commented 3 years ago

hmm I think that means the file's too large to fit in one string. max characters is .Machine$integer.max. you could try readChar(f, 1e6) instead

lime-n commented 3 years ago

Way too large to send onto here in GitHub without it crashing - Do you have any ideas on what the issue might be and how to solve it?

I am happy to send over one of these files

MichaelChirico commented 3 years ago

could you get gregexpr to work on that 1e6 subset?

the relevant part of the file is the first n characters where n is the gregexpr output

lime-n commented 3 years ago

Using the following:

x<-gregexpr("9c4a777e8b", readChar(fil[2], 1e6))[[1]]

Here is the output:

structure(c(1L, 505L, 1011L, 1556L, 2086L, 2567L, 3074L, 3575L, 
4077L, 4572L, 5072L, 5579L, 6097L, 6608L, 7152L, 7667L, 8170L, 
8656L, 9195L, 9692L, 10195L, 10703L, 11210L, 11722L, 12261L, 
12759L, 13294L, 13786L, 14309L, 14809L, 15306L, 15849L, 16349L, 
16900L, 17414L, 17915L, 18426L, 18947L, 19463L, 19994L, 20505L, 
21007L, 21519L, 22039L, 22539L, 23028L, 23549L, 24058L, 24561L, 
25099L, 25603L, 26123L, 26633L, 27147L, 27647L, 28156L, 28660L, 
29185L, 29693L, 30208L, 30693L, 31252L, 31718L, 32218L, 32743L, 
33293L, 33821L, 34334L, 34815L, 35324L, 35834L, 36313L, 36824L, 
37333L, 37883L, 38402L, 38938L, 39423L, 39929L, 40447L, 40952L, 
41447L, 41939L, 42447L, 42969L, 43482L, 43970L, 44473L, 44985L, 
45495L, 46011L, 46528L, 47017L, 47520L, 48053L, 48563L, 49063L, 
49569L, 50074L, 50580L, 51077L, 51581L, 51973L, 52489L, 53035L, 
53584L, 54094L, 54579L, 55093L, 55595L, 56097L, 56585L, 57076L, 
57590L, 58110L, 58636L, 59131L, 59618L, 60143L, 60653L, 61163L, 
61654L, 62167L, 62669L, 63163L, 63668L, 64170L, 64667L, 65199L, 
65701L, 66195L, 66703L, 67215L, 67699L, 68179L, 68674L, 69196L, 
69696L, 70203L, 70718L, 71249L, 71731L, 72241L, 72779L, 73265L, 
73771L, 74311L, 74829L, 75311L, 75828L, 76328L, 76872L, 77368L, 
77882L, 78383L, 78883L, 79384L, 79900L, 80414L, 80940L, 81456L, 
81945L, 82458L, 82973L, 83472L, 83975L, 84474L, 84981L, 85466L, 
85971L, 86509L, 87052L, 87574L, 88085L, 88597L, 89098L, 89628L, 
90154L, 90641L, 91156L, 91650L, 92189L, 92674L, 93197L, 93756L, 
94270L, 94765L, 95265L, 95741L, 96251L, 96743L, 97234L, 97736L, 
98224L, 98736L, 99279L, 99786L, 100287L, 100827L, 101343L, 101895L, 
102410L, 102935L, 103440L, 103970L, 104480L, 104987L, 105532L, 
106015L, 106552L, 107080L, 107568L, 108085L, 108598L, 109116L, 
109616L, 110116L, 110608L, 111108L, 111603L, 112104L, 112605L, 
113120L, 113638L, 114178L, 114712L, 115208L, 115734L, 116265L, 
116779L, 117292L, 117813L, 118311L, 118822L, 119331L, 119831L, 
120336L, 120823L, 121342L, 121835L, 122333L, 122823L, 123331L, 
123835L, 124356L, 124859L, 125355L, 125885L, 126381L, 126894L, 
127425L, 127897L, 128397L, 128901L, 129418L, 129938L, 130455L, 
130971L, 131479L, 131973L, 132477L, 132981L, 133499L, 133995L, 
134503L, 135020L, 135511L, 136024L, 136511L, 137013L, 137520L, 
138012L, 138522L, 139014L, 139520L, 140035L, 140525L, 141067L, 
141573L, 142078L, 142599L, 143097L, 143627L, 144181L, 144656L, 
145162L, 145654L, 146176L, 146703L, 147246L, 147762L, 148258L, 
148769L, 149280L, 149770L, 150275L, 150819L, 151335L, 151847L, 
152340L, 152852L, 153367L, 153906L, 154458L, 154969L, 155471L, 
155985L, 156484L, 156989L, 157494L, 158035L, 158569L, 159079L, 
159589L, 160106L, 160599L, 161114L, 161613L, 162114L, 162602L, 
163118L, 163622L, 164147L, 164654L, 165149L, 165666L, 166149L, 
166670L, 167180L, 167694L, 168201L, 168729L, 169272L, 169776L, 
170275L, 170752L, 171261L, 171805L, 172321L, 172829L, 173337L, 
173837L, 174328L, 174826L, 175331L, 175831L, 176350L, 176848L, 
177370L, 177899L, 178382L, 178917L, 179428L, 179940L, 180450L, 
180975L, 181453L, 181973L, 182473L, 182987L, 183491L, 183992L, 
184495L, 184999L, 185520L, 186032L, 186532L, 187014L, 187534L, 
188067L, 188575L, 189141L, 189657L, 190167L, 190666L, 191207L, 
191751L, 192242L, 192750L, 193255L, 193787L, 194294L, 194828L, 
195312L, 195819L, 196323L, 196888L, 197411L, 197906L, 198418L, 
198910L, 199406L, 199908L, 200400L, 200922L, 201422L, 201923L, 
202422L, 202925L, 203416L, 203919L, 204410L, 204910L, 205408L, 
205913L, 206401L, 206915L, 207438L, 207967L, 208487L, 209009L, 
209513L, 210046L, 210541L, 211042L, 211551L, 212068L, 212564L, 
213060L, 213581L, 214065L, 214566L, 215071L, 215579L, 216094L, 
216598L, 217123L, 217634L, 218174L, 218696L, 219192L, 219700L, 
220187L, 220679L, 221173L, 221682L, 222219L, 222760L, 223276L, 
223747L, 224232L, 224741L, 225271L, 225775L, 226285L, 226823L, 
227353L, 227875L, 228363L, 228871L, 229385L, 229910L, 230426L, 
230911L, 231426L, 231925L, 232438L, 232971L, 233474L, 233984L, 
234513L, 235018L, 235521L, 236036L, 236547L, 237065L, 237570L, 
238067L, 238564L, 239065L, 239591L, 240093L, 240598L, 241114L, 
241620L, 242106L, 242586L, 243092L, 243610L, 244151L, 244651L, 
245155L, 245671L, 246206L, 246736L, 247235L, 247750L, 248249L, 
248761L, 249271L, 249790L, 250274L, 250801L, 251288L, 251790L, 
252314L, 252800L, 253307L, 253819L, 254310L, 254846L, 255372L, 
255866L, 256400L, 256923L, 257427L, 257925L, 258437L, 258941L, 
259447L, 259924L, 260472L, 260971L, 261480L, 261981L, 262517L, 
263082L, 263601L, 264101L, 264605L, 265143L, 265640L, 266139L, 
266635L, 267137L, 267631L, 268142L, 268659L, 269175L, 269714L, 
270254L, 270766L, 271274L, 271781L, 272285L, 272815L, 273325L, 
273816L, 274308L, 274818L, 275321L, 275874L, 276378L, 276894L, 
277415L, 277920L, 278435L, 278940L, 279432L, 279944L, 280437L, 
280954L, 281455L, 281955L, 282481L, 282974L, 283472L, 283990L, 
284497L, 284981L, 285488L, 286001L, 286538L, 287064L, 287571L, 
288078L, 288570L, 289036L, 289553L, 290032L, 290555L, 291046L, 
291547L, 292050L, 292558L, 293048L, 293565L, 294089L, 294581L, 
295075L, 295583L, 296090L, 296580L, 297092L, 297596L, 298100L, 
298589L, 299114L, 299660L, 300182L, 300662L, 301188L, 301699L, 
302191L, 302712L, 303231L, 303731L, 304227L, 304740L, 305242L, 
305757L, 306281L, 306792L, 307299L, 307814L, 308347L, 308864L, 
309388L, 309893L, 310379L, 310899L, 311406L, 311904L, 312402L, 
312897L, 313432L, 313966L, 314474L, 314999L, 315498L, 315986L, 
316510L, 317002L, 317525L, 318048L, 318552L, 319059L, 319544L, 
320064L, 320562L, 321071L, 321588L, 322088L, 322602L, 323091L, 
323607L, 324116L, 324616L, 325084L, 325615L, 326104L, 326617L, 
327093L, 327597L, 328104L, 328619L, 329149L, 329651L, 330196L, 
330705L, 331226L, 331736L, 332241L, 332776L, 333304L, 333820L, 
334346L, 334846L, 335370L, 335898L, 336421L, 336925L, 337431L, 
337922L, 338422L, 338924L, 339431L, 339927L, 340453L, 340959L, 
341498L, 341987L, 342458L, 342974L, 343482L, 343987L, 344483L, 
344981L, 345484L, 346009L, 346512L, 347013L, 347511L, 348018L, 
348517L, 349040L, 349558L, 350057L, 350561L, 351066L, 351581L, 
352110L, 352618L, 353109L, 353634L, 354136L, 354663L, 355147L, 
355635L, 356155L, 356673L, 357165L, 357666L, 358191L, 358712L, 
359226L, 359752L, 360294L, 360801L, 361304L, 361794L, 362289L, 
362797L, 363299L, 363785L, 364295L, 364803L, 365321L, 365865L, 
366412L, 366916L, 367415L, 367932L, 368441L, 368949L, 369440L, 
369959L, 370459L, 370977L, 371493L, 372002L, 372537L, 373067L, 
373593L, 374123L, 374663L, 375158L, 375707L, 376208L, 376729L, 
377255L, 377763L, 378280L, 378781L, 379287L, 379792L, 380303L, 
380803L, 381302L, 381834L, 382337L, 382859L, 383387L, 383904L, 
384420L, 384933L, 385440L, 385931L, 386483L, 387002L, 387502L, 
388013L, 388503L, 389021L, 389539L, 390076L, 390595L, 391107L, 
391611L, 392109L, 392645L, 393186L, 393695L, 394214L, 394713L, 
395239L, 395766L, 396257L, 396732L, 397239L, 397745L, 398263L, 
398810L, 399312L, 399819L, 400362L, 400881L, 401399L, 401896L, 
402415L, 402913L, 403437L, 403961L, 404483L, 404986L, 405487L, 
406011L, 406494L, 406985L, 407493L, 408049L, 408560L, 409067L, 
409585L, 410091L, 410590L, 411093L, 411622L, 412139L, 412625L, 
413134L, 413696L, 414212L, 414693L, 415222L, 415692L, 416197L, 
416713L, 417204L, 417712L, 418234L, 418706L, 419204L, 419767L, 
420277L, 420782L, 421314L, 421818L, 422314L, 422858L, 423404L, 
423886L, 424378L, 424888L, 425400L, 425899L, 426406L, 426913L, 
427446L, 427932L, 428449L, 428936L, 429424L, 429933L, 430433L, 
430952L, 431463L, 432006L, 432524L, 433016L, 433512L, 434051L, 
434555L, 435078L, 435602L, 436101L, 436590L, 437083L, 437571L, 
438069L, 438571L, 439099L, 439610L, 440117L, 440598L, 441121L, 
441635L, 442153L, 442658L, 443163L, 443678L, 444197L, 444716L, 
445236L, 445736L, 446242L, 446759L, 447266L, 447769L, 448273L, 
448773L, 449279L, 449790L, 450294L, 450826L, 451391L, 451941L, 
452468L, 452993L, 453505L, 454019L, 454547L, 455068L, 455580L, 
456090L, 456581L, 457081L, 457579L, 458104L, 458614L, 459115L, 
459623L, 460136L, 460636L, 461171L, 461672L, 462198L, 462715L, 
463228L, 463731L, 464236L, 464767L, 465285L, 465798L, 466312L, 
466832L, 467312L, 467863L, 468380L, 468897L, 469393L, 469902L, 
470391L, 470891L, 471395L, 471886L, 472402L, 472946L, 473458L, 
473991L, 474488L, 474980L, 475482L, 475994L, 476486L, 477002L, 
477513L, 478017L, 478523L, 479087L, 479601L, 480098L, 480620L, 
481130L, 481645L, 482152L, 482654L, 483153L, 483636L, 484144L, 
484645L, 485162L, 485654L, 486161L, 486663L, 487167L, 487698L, 
488251L, 488753L, 489271L, 489790L, 490292L, 490812L, 491298L, 
491798L, 492348L, 492871L, 493394L, 493920L, 494451L, 494950L, 
495445L, 495995L, 496528L, 497028L, 497546L, 498055L, 498562L, 
499062L, 499616L, 500114L, 500616L, 501104L, 501590L, 502094L, 
502595L, 503123L, 503598L, 504097L, 504591L, 505084L, 505596L, 
506119L, 506625L, 507136L, 507652L, 508174L, 508689L, 509149L, 
509644L, 510185L, 510726L, 511215L, 511733L, 512239L, 512730L, 
513271L, 513787L, 514292L, 514782L, 515284L, 515785L, 516275L, 
516807L, 517315L, 517854L, 518340L, 518845L, 519359L, 519850L, 
520355L, 520863L, 521380L, 521879L, 522395L, 522915L, 523419L, 
523941L, 524470L, 524996L, 525518L, 526024L, 526529L, 527066L, 
527630L, 528121L, 528629L, 529134L, 529640L, 530210L, 530742L, 
531270L, 531783L, 532313L, 532830L, 533338L, 533824L, 534333L, 
534839L, 535293L, 535789L, 536304L, 536818L, 537384L, 537905L, 
538402L, 538938L, 539450L, 539987L, 540479L, 540989L, 541535L, 
542044L, 542552L, 543114L, 543623L, 544105L, 544639L, 545204L, 
545710L, 546192L, 546691L, 547176L, 547699L, 548208L, 548707L, 
549227L, 549749L, 550254L, 550765L, 551253L, 551764L, 552264L, 
552771L, 553277L, 553787L, 554287L, 554815L, 555322L, 555811L, 
556333L, 556834L, 557367L, 557871L, 558372L, 558912L, 559415L, 
559953L, 560455L, 560956L, 561475L, 561991L, 562504L, 563020L, 
563532L, 564039L, 564538L, 565068L, 565592L, 566095L, 566596L, 
567069L, 567597L, 568105L, 568594L, 569107L, 569627L, 570132L, 
570633L, 571180L, 571684L, 572180L, 572679L, 573180L, 573663L, 
574141L, 574655L, 575151L, 575626L, 576159L, 576682L, 577178L, 
577653L, 578148L, 578683L, 579194L, 579711L, 580209L, 580708L, 
581209L, 581711L, 582239L, 582729L, 583249L, 583767L, 584248L, 
584755L, 585270L, 585754L, 586281L, 586804L, 587314L, 587818L, 
588303L, 588804L, 589311L, 589850L, 590380L, 590938L, 591449L, 
591960L, 592448L, 592938L, 593465L, 593983L, 594539L, 595066L, 
595493L, 596017L, 596514L, 597006L, 597508L, 597942L, 598420L, 
598913L, 599442L, 599975L, 600484L, 600989L, 601487L, 601976L, 
602461L, 602966L, 603476L, 604016L, 604535L, 605024L, 605539L, 
606060L, 606566L, 607051L, 607594L, 608079L, 608570L, 609072L, 
609596L, 610118L, 610634L, 611154L, 611693L, 612203L, 612699L, 
613210L, 613693L, 614182L, 614705L, 615209L, 615717L, 616221L, 
616762L, 617275L, 617787L, 618294L, 618785L, 619270L, 619767L, 
620273L, 620771L, 621281L, 621791L, 622299L, 622821L, 623319L, 
623811L, 624355L, 624869L, 625428L, 625936L, 626442L, 626949L, 
627443L, 627950L, 628446L, 628966L, 629481L, 629982L, 630496L, 
630998L, 631542L, 632042L, 632545L, 633057L, 633584L, 634069L, 
634585L, 635067L, 635577L, 636074L, 636585L, 637100L, 637600L, 
638095L, 638604L, 639135L, 639635L, 640133L, 640651L, 641173L, 
641738L, 642221L, 642727L, 643213L, 643727L, 644228L, 644757L, 
645273L, 645777L, 646286L, 646802L, 647315L, 647819L, 648329L, 
648841L, 649346L, 649898L, 650452L, 650965L, 651474L, 651979L, 
652501L, 653012L, 653526L, 653999L, 654537L, 655045L, 655550L, 
656089L, 656624L, 657141L, 657691L, 658235L, 658735L, 659248L, 
659754L, 660254L, 660776L, 661252L, 661783L, 662293L, 662783L, 
663268L, 663764L, 664271L, 664788L, 665315L, 665808L, 666295L, 
666834L, 667354L, 667845L, 668346L, 668859L, 669416L, 669908L, 
670411L, 670916L, 671393L, 671944L, 672452L, 672949L, 673460L, 
673973L, 674505L, 675006L, 675513L, 676027L, 676533L, 677040L, 
677535L, 678075L, 678611L, 679108L, 679624L, 680168L, 680654L, 
681151L, 681642L, 682163L, 682669L, 683161L, 683671L, 684184L, 
684691L, 685212L, 685715L, 686227L, 686729L, 687278L, 687805L, 
688312L, 688835L, 689332L, 689844L, 690349L, 690893L, 691392L, 
691894L, 692416L, 692923L, 693425L, 693932L, 694453L, 694973L, 
695464L, 695986L, 696516L, 697022L, 697540L, 698045L, 698617L, 
699133L, 699617L, 700124L, 700638L, 701145L, 701645L, 702172L, 
702673L, 703178L, 703662L, 704180L, 704694L, 705169L, 705703L, 
706205L, 706684L, 707210L, 707707L, 708206L, 708725L, 709226L, 
709743L, 710260L, 710774L, 711289L, 711777L, 712295L, 712802L, 
713290L, 713803L, 714326L, 714850L, 715368L, 715873L, 716366L, 
716923L, 717426L, 717912L, 718430L, 718924L, 719427L, 719951L, 
720442L, 720929L, 721423L, 721980L, 722498L, 723009L, 723500L, 
723980L, 724475L, 724979L, 725505L, 726029L, 726545L, 727065L, 
727585L, 728080L, 728613L, 729112L, 729618L, 730125L, 730655L, 
731173L, 731638L, 732142L, 732664L, 733166L, 733671L, 734184L, 
734690L, 735178L, 735688L, 736188L, 736692L, 737195L, 737720L, 
738236L, 738764L, 739267L, 739763L, 740276L, 740768L, 741275L, 
741710L, 742265L, 742754L, 743242L, 743735L, 744262L, 744776L, 
745280L, 745822L, 746344L, 746844L, 747367L, 747857L, 748363L, 
748869L, 749371L, 749857L, 750333L, 750842L, 751374L, 751890L, 
752384L, 752888L, 753392L, 753906L, 754416L, 754918L, 755425L, 
755955L, 756457L, 756948L, 757484L, 757990L, 758512L, 759003L, 
759517L, 760057L, 760586L, 761088L, 761633L, 762135L, 762635L, 
763162L, 763669L, 764169L, 764657L, 765158L, 765669L, 766193L, 
766701L, 767232L, 767720L, 768234L, 768742L, 769208L, 769711L, 
770252L, 770750L, 771256L, 771772L, 772283L, 772833L, 773324L, 
773811L, 774294L, 774786L, 775332L, 775842L, 776346L, 776851L, 
777344L, 777875L, 778383L, 778890L, 779413L, 779929L, 780426L, 
780929L, 781413L, 781922L, 782433L, 782936L, 783450L, 783951L, 
784458L, 784992L, 785453L, 785967L, 786517L, 787008L, 787518L, 
788029L, 788542L, 789025L, 789545L, 790050L, 790558L, 791072L, 
791579L, 792089L, 792606L, 793138L, 793655L, 794170L, 794687L, 
795179L, 795683L, 796201L, 796735L, 797246L, 797763L, 798280L, 
798767L, 799274L, 799776L, 800291L, 800811L, 801316L, 801837L, 
802317L, 802828L, 803356L, 803859L, 804370L, 804872L, 805386L, 
805908L, 806474L, 806985L, 807530L, 808034L, 808542L, 809080L, 
809578L, 810080L, 810600L, 811105L, 811602L, 812075L, 812589L, 
813094L, 813602L, 814102L, 814636L, 815140L, 815650L, 816188L, 
816668L, 817190L, 817707L, 818206L, 818706L, 819270L, 819807L, 
820338L, 820818L, 821347L, 821887L, 822400L, 822904L, 823419L, 
823909L, 824417L, 824899L, 825388L, 825884L, 826407L, 826911L, 
827419L, 827929L, 828446L, 828965L, 829492L, 830002L, 830519L, 
831021L, 831528L, 832005L, 832529L, 833027L, 833517L, 834025L, 
834530L, 835022L, 835538L, 836030L, 836561L, 837065L, 837600L, 
838120L, 838662L, 839202L, 839704L, 840223L, 840739L, 841246L, 
841778L, 842297L, 842812L, 843328L, 843825L, 844377L, 844842L, 
845327L, 845840L, 846368L, 846873L, 847404L, 847953L, 848480L, 
848983L, 849475L, 850016L, 850539L, 851049L, 851568L, 852087L, 
852609L, 853122L, 853625L, 854126L, 854628L, 855126L, 855638L, 
856158L, 856691L, 857200L, 857686L, 858170L, 858686L, 859182L, 
859721L, 860233L, 860733L, 861260L, 861769L, 862279L, 862782L, 
863274L, 863805L, 864316L, 864827L, 865327L, 865868L, 866368L, 
866873L, 867385L, 867887L, 868405L, 868912L, 869435L, 869934L, 
870450L, 870973L, 871457L, 871966L, 872477L, 873007L, 873516L, 
874032L, 874549L, 875053L, 875536L, 876057L, 876601L, 877102L, 
877600L, 878103L, 878656L, 879172L, 879716L, 880254L, 880787L, 
881290L, 881795L, 882294L, 882790L, 883297L, 883826L, 884348L, 
884872L, 885370L, 885894L, 886405L, 886913L, 887430L, 887916L, 
888439L, 888938L, 889448L, 889975L, 890501L, 890984L, 891504L, 
892008L, 892526L, 893024L, 893523L, 894030L, 894558L, 895073L, 
895567L, 896109L, 896625L, 897129L, 897651L, 898158L, 898665L, 
899191L, 899701L, 900233L, 900739L, 901233L, 901740L, 902213L, 
902755L, 903253L, 903758L, 904289L, 904787L, 905273L, 905810L, 
906305L, 906805L, 907314L, 907821L, 908314L, 908815L, 909334L, 
909836L, 910347L, 910858L, 911362L, 911851L, 912354L, 912852L, 
913374L, 913889L, 914396L, 914923L, 915460L, 915953L, 916451L, 
916933L, 917443L, 917953L, 918458L, 918989L, 919496L, 919986L, 
920487L, 921015L, 921513L, 922055L, 922568L, 923098L, 923626L, 
924165L, 924684L, 925208L, 925750L, 926257L, 926792L, 927351L, 
927839L, 928359L, 928869L, 929391L, 929888L, 930387L, 930873L, 
931374L, 931883L, 932427L, 932914L, 933432L, 933932L, 934431L, 
934954L, 935452L, 935953L, 936454L, 936963L, 937490L, 937989L, 
938496L, 939003L, 939524L, 940034L, 940541L, 941050L, 941562L, 
942067L, 942597L, 943098L, 943601L, 944104L, 944630L, 945152L, 
945656L, 946158L, 946641L, 947114L, 947594L, 948105L, 948604L, 
949103L, 949590L, 950109L, 950632L, 951115L, 951606L, 952124L, 
952631L, 953147L, 953660L, 954180L, 954684L, 955189L, 955681L, 
956188L, 956686L, 957216L, 957729L, 958231L, 958740L, 959262L, 
959764L, 960280L, 960779L, 961277L, 961794L, 962290L, 962782L, 
963315L, 963820L, 964306L, 964809L, 965296L, 965808L, 966316L, 
966824L, 967308L, 967845L, 968346L, 968861L, 969369L, 969886L, 
970415L, 970917L, 971418L, 971925L, 972423L, 972945L, 973483L, 
973987L, 974494L, 975046L, 975548L, 976061L, 976566L, 977089L, 
977599L, 978098L, 978651L, 979152L, 979665L, 980167L, 980685L, 
981205L, 981730L, 982227L, 982735L, 983235L, 983796L, 984295L, 
984812L, 985318L, 985834L, 986339L, 986861L, 987393L, 987911L, 
988390L, 988908L, 989409L, 989925L, 990407L, 990910L, 991439L, 
991938L, 992420L, 992977L, 993460L, 993973L, 994490L, 994998L, 
995532L, 996030L, 996529L, 997029L, 997530L, 998032L, 998535L, 
999043L, 999544L), match.length = c(10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 10L, 
10L, 10L, 10L, 10L, 10L))
lime-n commented 3 years ago

I believe that I know what the issue is here!

After piecing together what you have mentioned and asked me to do I have noticed after looking at the verbose text again - That it automatically detected it as sep=",", however it is clearly a tab-delimited file given by the previous output from readChar(f, 500). Michael you have taught me lots in this discussion towards debugging situations like this, so I thank you a lot for this as it will be tremendously helpful when I work with big data again.

Though I am interested to know why it suddenly shifted from detecting it as "\t" to ",", and whether there is a better method that detects one over the other? perhaps by column sizes in my situation?

MichaelChirico commented 3 years ago

oh nice, I had missed that! It seems the sep auto-detection logic god thrown off then, in that case.

we would have to have some minimal version of the file to be able to dive deeply into why , was chosen over \t for this particular file... the verbose output here mentions:

[06] Detect separator, quoting rule, and ncolumns
  Detecting sep automatically ...
  sep=','  with 100 lines of 3 fields using quote rule 0
  Detected 3 columns on line 1. This line is either column names or first data row. Line starts as: <<9c4a777e8b    453931765   Animalia    >>
  Quote rule picked = 0
  fill=false and the most number of columns found is 3

IIRC there's a plurality rule where consistency of the lines is checked by various quote rule/sep combinations, with a tie going to the seps in order and I think , comes before \t. So maybe that's happening here -- , and \t are tied for suitability, but ties default to ,.

Maybe it would be useful for verbose=TRUE to report the "second place" score as well to give some insight that there's a close competition for this file.