search

ssc-oscar / lookup

A mirror of bitbucket.org/swcs/lookup
1 stars 4 forks source link

Meaning of "Split loop" #32

Closed k----n closed 3 years ago

k----n commented 3 years ago

I've recently come across this for b2c lookup:

Split loop at ~/lookup/getValues.perl line 165, <STDIN> line 1135.

According to https://www.perlmonks.org/?node_id=232081 it's a runtime error.

audrism commented 3 years ago

please run git pull.

Perl does not like statements like:

for my $vv (split(/;/, $res, -1)){
}
k----n commented 3 years ago

I pulled the latest changes from https://bitbucket.org/swsc/lookup/

Still getting the error but without <STDIN> line:

Split loop at ~/lookup/getValues.perl line 175.
audrism commented 3 years ago

My bad, forgot to push...

k----n commented 3 years ago

Thanks, I pulled https://bitbucket.org/swsc/lookup/commits/a2e7fb7e5ad03398682a66fdfc6f562b4c185b75 and the issue still seems to persist with this error:

Split loop at ~/lookup/getValues.perl line 175.
audrism commented 3 years ago

Just to confirm: line 175 is

my @vvs = split(/;/, $res, -1);

what is input to b2c?

k----n commented 3 years ago

Yes it's:

      my @vvs = split(/;/, $res, -1);

Input to b2c is the list of blob sha1s : b_000.txt

Run with: cat b_000.txt | ~/lookup/getValues -f b2c

It produces 12 of the split loop errors.

audrism commented 3 years ago

I suspect perl has limits on how many items it can produce in a split and some of the blobs are in millions of commits:

grep -Ff b_000.txt ~/lookup/woc.pm 

  "e69de29bb2d1d6434b8b29ae775ad8c2e48c5391" => 7826798, #\n 
  "8b137891791fe96927ad78e64b0aad7bded08bdc" => 957235, #\n\n
  "25bf17fc5aaabd17402e77a2b16f95fbea7310d2" => 100000, # more than that "18"
  "0637a088a01e8ddab3bf3fa98dbe804cbde1a0dc" => 100000, # more than that "[]"

I wrote a fix in ~audris/lookup, but need to make sure it is correct before checking in.