Bad things [still] happen during server rebalance

[ivulfson]

I have a script that does a batch get() of 100k keys by id from the bucket. If you start rebalancing the cluster when the script is running, this happens:

Unrecognized flags 0x2. Assuming raw at ./dump.pl line 49.
Unrecognized flags 0x2. Assuming raw at ./dump.pl line 49.
Unrecognized flags 0x2. Assuming raw at ./dump.pl line 49.
Unrecognized flags 0x2. Assuming raw at ./dump.pl line 49.
Unrecognized flags 0x2. Assuming raw at ./dump.pl line 49.
encountered object 'Couchbase::Document=ARRAY(0x3fba26f0)', but neither allow_blessed nor convert_blessed settings are enabled at ./dump.pl line 51.
Attempt to free unreferenced scalar: SV 0x43fd0fc0, Perl interpreter: 0x10c0010 at ./dump.pl line 51.
Attempt to free unreferenced scalar: SV 0x43f83d10, Perl interpreter: 0x10c0010 during global destruction.
Attempt to free unreferenced scalar: SV 0x43f86788, Perl interpreter: 0x10c0010 during global destruction.
Attempt to free unreferenced scalar: SV 0x43f86d88, Perl interpreter: 0x10c0010 during global destruction.
Attempt to free unreferenced scalar: SV 0x43f87388, Perl interpreter: 0x10c0010 during global destruction.
Attempt to free unreferenced scalar: SV 0x43f89e00, Perl interpreter: 0x10c0010 during global destruction.
Attempt to free unreferenced scalar: SV 0x43f8a400, Perl interpreter: 0x10c0010 during global destruction.
Attempt to free unreferenced scalar: SV 0x43f8aa00, Perl interpreter: 0x10c0010 during global destruction.
Attempt to free unreferenced scalar: SV 0x43f8c548, Perl interpreter: 0x10c0010 during global destruction.

and then it dies. I remember there was a similar issue in libcouchbase 2.0.7 and 2.1.3, but i'm using libcouchbase 2.4.2, perl-Couchbase 2.0.0_2, and hitting Couchbase server 3.0.0.

Here's the [ugly] yet relevant perl code with line numbers:

    36      my $i = 0;
    37      while (my @batch = splice(@keys, 0, $cfg->{batch_size})) {
    38          $i += @batch;
    39          print "$i\n";
    40  
    41          my @docs = map { Couchbase::Document->new($_) } @batch;
    42  
    43          while (@docs) {
    44              my $batch = $CB->batch;
    45              $batch->get($_) for @docs;
    46  
    47              my @redo_docs = ();
    48              my %errors    = ();
    49              while (my $doc = $batch->wait_one) {
    50                  if ($doc->is_ok) {
    51                      print $fh $doc->id . "\t" . $JSON->encode($doc->value) . "\n";
    52                  } else {
    53                      my $error = $doc->errnum;
    54                      warn "Error getting " . $doc->id . ", will redo (other similar errors are silenced): $error\n" if !$errors{$error}++;
    55                      push @redo_docs, $doc;
    56                  }
    57              }
    58  
    59              @docs = @redo_docs;
    60              if (@redo_docs) {
    61                  foreach my $error (sort keys %errors) {
    62                      warn "Found $errors{$error} errors: $error\n";
    63                  }
    64                  print "sleeping...\n";
    65                  sleep 3;
    66              }
    67          }
    68      }

[ivulfson]

Agreed. Not the client library issue. Awaiting 3.0.1 server.

[ivulfson]

Thank you very much for your help in diagnosing this. :)

[ivulfson]

Couchbase Server 3.0.1 has been released, and this issue is indeed fixed. Thank you!