breser / git2consul

Mirrors the contents of a git repository into Consul KVs.
Other
763 stars 164 forks source link

Files not in source_root are killing my consul API. #188

Open ewah opened 4 years ago

ewah commented 4 years ago

files not the source_root will go into check pending [maybe this isn't a good place]

    if (branch.source_root && record.path.indexOf(branch.source_root) !== 0) {
      return check_pending();
    };

and the pending_records is ALWAYS decremented [it should not be]

--pending_records

when it hits zero [and it will eventually because pending never got incremented] it calls the callback of process_records

_.once(cb((errors_seen.length > 0) ? errors_seen : null));

the errorless callback will setLastProcessedRef for each file that is not in

        exports.setLastProcessedRef(branch, ref, function(err) {
          return cb(err);
        });

Each setLastProcessedRef will make an api call.

exports.setLastProcessedRef = function(branch, ref, cb) {
  write_content_to_consul(create_key_name(branch, branch.name + '.ref', true), ref, cb);
};

NOTE: though well-intented, the _.once() two blocks up doesn't actually stop this call.


I have a quick local fix limiting the files coming in. within lib/git/commands.js (and relevant caller)

exports.listAllFiles = function(cwd, source_root, cb) {
  run_command('git ls-tree --name-status -r HEAD', cwd, function(err, output) {
    /* istanbul ignore if */
    if (err) return cb(err);

    var records = [];
    var files = output.split('\n');
    files.forEach(function(file) {
      // this condition added.
      if (source_root && file.indexOf(source_root) !== 0) {
        return;
      };

      records.push({'type': 'M', 'path': file});
    });

    cb(null, records);
  });
};

But, I'm hoping someone has fixed/will fix the pending counter.

ewah commented 4 years ago

This is a good idea also.

    if (branch.source_root && record.path.indexOf(branch.source_root) !== 0) {
      // return check_pending();
      return;
    };