unexpected duplicate zip parts for zipped_moab_versions rows

[jmartin-sul]

A ZippedMoabVersion represents a zipped version directory from a moab, for a specific druid, on a specific cloud endpoint. Since each cloud endpoint only stores one logical copy of a moab-version directory for a given druid, we should not see more than one ZipPart of a given suffix for a given ZippedMoabVersion.

Ultimately, we should add a unique constraint to zip_parts on combo of zipped_moab_version_id and suffix. But...

Maybe I'm constructing the wrong query for this, but there appears to be a disconcerting amount of this sort of thing to clean up on prod before we can apply such a constraint:

pry(main)> zmvs_with_dupe_suffixes = ZipPart.group(:zipped_moab_version_id, :suffix).having('count(id) > 1').pluck(:zipped_moab_version_id, :suffix)
=> [[1, ".zip"],
 [3, ".zip"],
 [5, ".zip"],
...
 [19656334, ".zip"]]
pry(main)> zmvs_with_dupe_suffixes.size
=> 5904
pry(main)> ZippedMoabVersion.count
=> 16727610
pry(main)> ZipPart.group(:zipped_moab_version_id, :suffix).having('count(id) > 1').select(:zipped_moab_version_id, :suffix).to_sql
=> "SELECT \"zip_parts\".\"zipped_moab_version_id\", \"zip_parts\".\"suffix\" FROM \"zip_parts\" GROUP BY \"zip_parts\".\"zipped_moab_version_id\", \"zip_parts\".\"suffix\" HAVING (count(id) > 1)"

the sql looks right to me, but a second set of eyes would be good:

SELECT zip_parts.zipped_moab_version_id, zip_parts.suffix
FROM zip_parts
GROUP BY zip_parts.zipped_moab_version_id, zip_parts.suffix
HAVING (count(id) > 1)

5904 out of 16727610 isn't insane, but it's still significant, esp since we're not sure how this happens.

the only automatable remediation i can think of to start would be to:

• iterate over the results of the above query
  • for each combination, if there's an ok version of the ZipPart for the given suffix/ZMV combo, check the cloud endpoint to see that it's actually still present.
  • if it is, keep the ok row and destroy the redundant ZipParts

as a start, maybe we should add an ActiveRecord validation to prevent more occurrences of this before remediating?

i was worried that this would be blocked by #1530, but i don't think that schema change will cause problems here, since the preserved_objects and complete_moabs never come into play directly.

discovered while investigating #1410.

I could imagine splitting this ticket into three (sequenced) sub tickets:

• [ ] add an ActiveRecord validation to prevent further occurrences
• [ ] remediate the existing bad data (could be done before the validation was added, but risks having more bad data accrue till we figure the issue out and stop it from growing)
• [ ] add the unique constraint (blocked by remediating bad data)

[mjgiarlo]

Get the IDs of unreplicated ZipParts with duplicate OK ZipParts:

zmvs_with_dupe_suffixes
  .select { |zmv_id, suffix| ZipPart.find_by(zipped_moab_version_id: zmv_id, suffix: suffix, status: 'ok') }
  .map { |zmv_id, suffix| ZipPart.find_by(zipped_moab_version_id: zmv_id, suffix: suffix, status: 'unreplicated') }
  .compact
  .map(&:id)
  .sort

FWIW, there are 5,242 of these.

[mjgiarlo]

Needs further analysis: what's going on with the other 662 zip parts. Perhaps there's a different pair of statuses than 1 OK and 1 unrepl for those?

[mjgiarlo]

1 of the 662 is:

ZipPart.where(zipped_moab_version_id: 8340588)
=> [#<ZipPart:0x00000000055b3678
  id: 8401506,
  size: 193297171,
  zipped_moab_version_id: 8340588,
  created_at: Wed, 14 Nov 2018 18:29:37 UTC +00:00,
  updated_at: Wed, 14 Nov 2018 18:29:37 UTC +00:00,
  md5: "ab6ec2ae1e2b1828809dfc6d45d1bea7",
  create_info:
   "{:zip_cmd=>\"zip -r0X -s 10g /sdr-transfers/bp/165/vv/7725/bp165vv7725.v0001.zip bp165vv7725/v0001\", :zip_version=>\"Zip 3.0 (July 5th 2008)\"}",
  parts_count: 3,
  suffix: ".zip",
  status: "unreplicated",
  last_existence_check: nil,
  last_checksum_validation: nil>,
 #<ZipPart:0x00000000055b3510
  id: 9036758,
  size: 193297171,
  zipped_moab_version_id: 8340588,
  created_at: Thu, 07 Mar 2019 21:56:22 UTC +00:00,
  updated_at: Thu, 07 Mar 2019 21:56:22 UTC +00:00,
  md5: "ab6ec2ae1e2b1828809dfc6d45d1bea7",
  create_info:
   "{:zip_cmd=>\"zip -r0X -s 10g /sdr-transfers/bp/165/vv/7725/bp165vv7725.v0001.zip bp165vv7725/v0001\", :zip_version=>\"Zip 3.0 (July 5th 2008)\"}",
  parts_count: 1,
  suffix: ".zip",
  status: "unreplicated",
  last_existence_check: nil,
  last_checksum_validation: nil>]

So, both unreplicated.

[mjgiarlo]

Here are the unique ZipPart dup statuses:

> zmvs_with_dupe_suffixes.map { |zmv_id, suffix| ZipPart.where(zipped_moab_version_id: zmv_id, suffix: suffix).pluck(:status) }.sort.uniq
[["ok", "ok"],
 ["ok", "ok", "unreplicated"],
 ["ok", "ok", "unreplicated", "unreplicated"],
 ["ok", "unreplicated"],
 ["ok", "unreplicated", "ok"],
 ["ok", "unreplicated", "unreplicated"],
 ["ok", "unreplicated", "unreplicated", "ok"],
 ["ok", "unreplicated", "unreplicated", "unreplicated"],
 ["unreplicated", "ok"],
 ["unreplicated", "ok", "ok"],
 ["unreplicated", "unreplicated"],
 ["unreplicated", "unreplicated", "ok"],
 ["unreplicated", "unreplicated", "unreplicated", "ok"],
 ["unreplicated", "unreplicated", "unreplicated", "unreplicated", "ok"],
 ["unreplicated", "unreplicated", "unreplicated", "unreplicated", "unreplicated", "unreplicated", "unreplicated", "unreplicated", "ok"]]

[mjgiarlo]

There are 351 ZipParts with > 2 statuses:

> zmvs_with_dupe_suffixes.select { |zmv_id, suffix| ZipPart.where(zipped_moab_version_id: zmv_id, suffix: suffix).count > 2 }.count
=> 351

[mjgiarlo]

There are 4,891 ZipParts with exactly 2 statuses, where one is OK and the other is unreplicated:

> zmvs_with_dupe_suffixes.select { |zmv_id, suffix| ZipPart.where(zipped_moab_version_id: zmv_id, suffix: suffix).pluck(:status) == ['ok', 'unreplicated']}.count
=> 4054
> zmvs_with_dupe_suffixes.select { |zmv_id, suffix| ZipPart.where(zipped_moab_version_id: zmv_id, suffix: suffix).pluck(:status) == ['unreplicated', 'ok']}.count
=> 837

[mjgiarlo]

There are 661 ZipParts with exactly 2 status and BOTH are OK:

> zmvs_with_dupe_suffixes.select { |zmv_id, suffix| ZipPart.where(zipped_moab_version_id: zmv_id, suffix: suffix).pluck(:status) == ['ok', 'ok']}.count
=> 661

[mjgiarlo]

This may be interesting, @jmartin-sul ...here's the breakdown of the 5,904 offending ZipParts (ZipParts which have duplicate rows in terms of ZMV ID and suffix):

Number	Description
4,891	1 OK ZipPart, 1 unreplicated ZipPart
661	2 OK ZipParts
351	3 or more ZipParts (combination of various OKs and unreplicateds
1	2 unreplicated ZipParts

For the 4,891 ZipParts, the remediation outlined above should work just fine. How should we deal with:

• Multiple OKs?
  • Pick one OK and remove the other OKs and unreplicateds? If so, what are the decision criteria to pick an OK?
• Two or more unreplicateds and no OKs?
• Two or more unreplicateds with one OK?
  • Remove the multiple unreplicateds?

@jmartin-sul your advice? or should we pair?

[jmartin-sul]

This may be interesting, @jmartin-sul ...here's the breakdown of the 5,904 offending ZipParts (ZipParts which have duplicate rows in terms of ZMV ID and suffix): [snip because apparently you can't quote table formatting]

For the 4,891 ZipParts, the remediation outlined above should work just fine. How should we deal with:

• Multiple OKs?

  • Pick one OK and remove the other OKs and unreplicateds? If so, what are the decision criteria to pick an OK?

yeah, i think just pick one of the oks? would be good to see if the duplicate-seeming parts were otherwise equal, i.e. do md5/size/parts_count/create_info agree across the dupes, and does parts_count agree with the other parts for the parent ZMV (if any)? and with the cloud copy?

if we were unsure about the cloud copy, we could always pull it down and see if it unzips (would be a bit of a pain to checksum validate it, but there are options for that).

• Two or more unreplicateds and no OKs?

i think here i'd see if the cloud copy looked good, and if it did, i'd update the DB to reflect that (i.e. to have one ok copy for the suffix). and if it looks iffy, we can consider removing the bad archive copy (and associated ZipParts and ZippedMoabVersion), then re-running replication for the moab (which should just queue replication for the unreplicated version/endpoint combos).

• Two or more unreplicateds with one OK?

  • Remove the multiple unreplicateds?

i think just take the ok assuming all else looks good?

@jmartin-sul your advice? or should we pair?

another thought... this is a small enough set that maybe we should just loop over all the druids turned up by this investigation, and run some checks on each one. some possible checks to run:

• the body of the PartReplicationAuditJob#perform, which should check for part count inconsistencies and should confirm that any supposedly replicated parts are actually out there: https://github.com/sul-dlss/preservation_catalog/blob/bf423887d91457644bddeae3ca9a63bac747131f/app/jobs/part_replication_audit_job.rb#L15-L20
  • i just pushed a branch that removes the unreplicated filter from the existing part audit checks (to my slight surprise, it breaks no tests, but probably deserves a couple new tests): https://github.com/sul-dlss/preservation_catalog/compare/check-unreplicated-zip-parts
• i wonder whether we should also compare md5/size/parts_count/create_info between rows for a given set of possible dupes. i'm not sure whether that's worth it or how to best automate that. if discrepancies were few, there wouldn't be too much to think about.

so... maybe we should pair to discuss how to proceed?

[mjgiarlo]

zmvs_with_dupe_suffixes = ZipPart.group(:zipped_moab_version_id, :suffix).having('count(id) > 1').pluck(:zipped_moab_version_id, :suffix)

# NOTE: If useful to subdivide the dups by status, you can do that as follows:
#
# zmvs_one_ok_one_unrepl = zmvs_with_dupe_suffixes.select { |zmv_id, suffix| ZipPart.where(zipped_moab_version_id: zmv_id, suffix: suffix).pluck(:status).sort == ['ok', 'unreplicated']} # 4,000+
# zmvs_both_ok = zmvs_with_dupe_suffixes.select { |zmv_id, suffix| ZipPart.where(zipped_moab_version_id: zmv_id, suffix: suffix).pluck(:status) == ['ok', 'ok']} # ~1,000
# zmvs_three_or_more_statuses = zmvs_with_dupe_suffixes.select { |zmv_id, suffix| ZipPart.where(zipped_moab_version_id: zmv_id, suffix: suffix).count > 2 } # ~300
# zmvs_both_unrepl = zmvs_with_dupe_suffixes.select { |zmv_id, suffix| ZipPart.where(zipped_moab_version_id: zmv_id, suffix: suffix).pluck(:status) == ['unreplicated', 'unreplicated']} # 1

def remediate(zmv_dupes)
  zmv_dupes.each do |zmv_id, suffix|
    puts "analyzing zmv #{zmv_id} suffix #{suffix}"
    zmv = ZippedMoabVersion.find(zmv_id)
    druid = zmv.complete_moab.preserved_object.druid
    version = zmv.version
    endpoint =  zmv.zip_endpoint
    parts = zmv.zip_parts.where(suffix: suffix)

    unique_s3_keys = parts.map(&:s3_key).uniq
    if unique_s3_keys.size > 1
      warn "one or more replicated parts has different S3 keys! #{unique_s3_keys} for zmv #{zmv_id} / suffix #{suffix} (druid: #{druid}, version #{version}, endpoint: #{endpoint.endpoint_name}), skipping"
      next
    end

    s3_key = unique_s3_keys.first
    audit = endpoint.audit_class.new(zmv, AuditResults.new(druid, version, nil, 'manual_cloud_archive_audit'))

    if !audit.send(:bucket).object(s3_key).exists?
      warn "part does not exist in the cloud for #{s3_key} zmv #{zmv_id} / suffix #{suffix} (druid: #{druid}, version #{version}, endpoint: #{endpoint.endpoint_name}), skipping"
      next
    end

    cloud_size = audit.send(:bucket).object(s3_key).content_length
    cloud_md5, cloud_parts_count = audit.send(:bucket).object(s3_key).metadata.slice('checksum_md5', 'parts_count').values

    parts_in_order = parts.order('created_at DESC')
    most_recent_part_where_database_matches_cloud = parts_in_order.find do |part|
      part.size == cloud_size && part.md5 == cloud_md5 && part.parts_count == cloud_parts_count.to_i
    end

    if most_recent_part_where_database_matches_cloud.blank?
      warn "could not find a part whose metadata (md5/size/parts_count) matches all the same values in the cloud endpoint #{s3_key} zmv #{zmv_id} / suffix #{suffix} (druid: #{druid}, version #{version}, endpoint: #{endpoint.endpoint_name}), skipping"
      next
    end

    most_recent_part_where_database_matches_cloud.ok! unless most_recent_part_where_database_matches_cloud.ok?

    parts_in_order.each do |part|
      next if part.id == most_recent_part_where_database_matches_cloud.id
      puts "destroying ZipPart #{part.id} for zmv #{zmv_id} / suffix #{suffix} (druid: #{druid}, version #{version}, endpoint: #{endpoint.endpoint_name})"
      part.destroy!
    end
  end
end

[mjgiarlo]

The majority of the "one ok, one unreplicated" ZMV dupes agree on parts_count, size, and md5. That's 4,076 of the 4,569. For all of those, all replicated parts exist.

Here are the (493) problematic ones:

analyzing zmv 797299 suffix .z02
duplicate parts for zmv 797299 / suffix .z02 (druid: dy259gh7906, version 1, endpoint: aws_s3_west_2) disagree on md5, skipping
analyzing zmv 2562030 suffix .zip
duplicate parts for zmv 2562030 / suffix .zip (druid: fk847cz0044, version 1, endpoint: aws_s3_west_2) disagree on md5, skipping
analyzing zmv 3493580 suffix .zip
duplicate parts for zmv 3493580 / suffix .zip (druid: jq797jt1037, version 1, endpoint: aws_s3_west_2) disagree on md5, skipping
analyzing zmv 6441892 suffix .zip
duplicate parts for zmv 6441892 / suffix .zip (druid: fm001pk4253, version 1, endpoint: aws_s3_west_2) disagree on md5, skipping
analyzing zmv 7672435 suffix .zip
duplicate parts for zmv 7672435 / suffix .zip (druid: gr404gd5445, version 1, endpoint: aws_s3_west_2) disagree on md5, skipping
analyzing zmv 7853119 suffix .z03
duplicate parts for zmv 7853119 / suffix .z03 (druid: zg936mz3511, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7853119 suffix .z08
duplicate parts for zmv 7853119 / suffix .z08 (druid: zg936mz3511, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7853119 suffix .zip
duplicate parts for zmv 7853119 / suffix .zip (druid: zg936mz3511, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7856269 suffix .z01
duplicate parts for zmv 7856269 / suffix .z01 (druid: zg502tr5183, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7856269 suffix .z13
duplicate parts for zmv 7856269 / suffix .z13 (druid: zg502tr5183, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7856269 suffix .z14
duplicate parts for zmv 7856269 / suffix .z14 (druid: zg502tr5183, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7856269 suffix .z15
duplicate parts for zmv 7856269 / suffix .z15 (druid: zg502tr5183, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7856269 suffix .zip
duplicate parts for zmv 7856269 / suffix .zip (druid: zg502tr5183, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857115 suffix .z01
duplicate parts for zmv 7857115 / suffix .z01 (druid: yr010gy2124, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857115 suffix .z02
duplicate parts for zmv 7857115 / suffix .z02 (druid: yr010gy2124, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857115 suffix .z03
duplicate parts for zmv 7857115 / suffix .z03 (druid: yr010gy2124, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857115 suffix .z04
duplicate parts for zmv 7857115 / suffix .z04 (druid: yr010gy2124, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857115 suffix .z05
duplicate parts for zmv 7857115 / suffix .z05 (druid: yr010gy2124, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857115 suffix .z06
duplicate parts for zmv 7857115 / suffix .z06 (druid: yr010gy2124, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857115 suffix .z07
duplicate parts for zmv 7857115 / suffix .z07 (druid: yr010gy2124, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857115 suffix .z08
duplicate parts for zmv 7857115 / suffix .z08 (druid: yr010gy2124, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857115 suffix .z09
duplicate parts for zmv 7857115 / suffix .z09 (druid: yr010gy2124, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857115 suffix .z10
duplicate parts for zmv 7857115 / suffix .z10 (druid: yr010gy2124, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857115 suffix .z11
duplicate parts for zmv 7857115 / suffix .z11 (druid: yr010gy2124, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857115 suffix .z12
duplicate parts for zmv 7857115 / suffix .z12 (druid: yr010gy2124, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857115 suffix .z13
duplicate parts for zmv 7857115 / suffix .z13 (druid: yr010gy2124, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857115 suffix .z14
duplicate parts for zmv 7857115 / suffix .z14 (druid: yr010gy2124, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857115 suffix .z15
duplicate parts for zmv 7857115 / suffix .z15 (druid: yr010gy2124, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857115 suffix .z16
duplicate parts for zmv 7857115 / suffix .z16 (druid: yr010gy2124, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857115 suffix .z17
duplicate parts for zmv 7857115 / suffix .z17 (druid: yr010gy2124, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857115 suffix .z18
duplicate parts for zmv 7857115 / suffix .z18 (druid: yr010gy2124, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857115 suffix .z19
duplicate parts for zmv 7857115 / suffix .z19 (druid: yr010gy2124, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857115 suffix .z21
duplicate parts for zmv 7857115 / suffix .z21 (druid: yr010gy2124, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857115 suffix .z22
duplicate parts for zmv 7857115 / suffix .z22 (druid: yr010gy2124, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857115 suffix .z23
duplicate parts for zmv 7857115 / suffix .z23 (druid: yr010gy2124, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857115 suffix .z24
duplicate parts for zmv 7857115 / suffix .z24 (druid: yr010gy2124, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857115 suffix .z25
duplicate parts for zmv 7857115 / suffix .z25 (druid: yr010gy2124, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857115 suffix .z26
duplicate parts for zmv 7857115 / suffix .z26 (druid: yr010gy2124, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857115 suffix .z27
duplicate parts for zmv 7857115 / suffix .z27 (druid: yr010gy2124, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857115 suffix .zip
duplicate parts for zmv 7857115 / suffix .zip (druid: yr010gy2124, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857191 suffix .z03
duplicate parts for zmv 7857191 / suffix .z03 (druid: mx490qf1543, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857191 suffix .z12
duplicate parts for zmv 7857191 / suffix .z12 (druid: mx490qf1543, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857191 suffix .z13
duplicate parts for zmv 7857191 / suffix .z13 (druid: mx490qf1543, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857191 suffix .z14
duplicate parts for zmv 7857191 / suffix .z14 (druid: mx490qf1543, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857191 suffix .z15
duplicate parts for zmv 7857191 / suffix .z15 (druid: mx490qf1543, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857191 suffix .z16
duplicate parts for zmv 7857191 / suffix .z16 (druid: mx490qf1543, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857191 suffix .z17
duplicate parts for zmv 7857191 / suffix .z17 (druid: mx490qf1543, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857191 suffix .z18
duplicate parts for zmv 7857191 / suffix .z18 (druid: mx490qf1543, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857191 suffix .z19
duplicate parts for zmv 7857191 / suffix .z19 (druid: mx490qf1543, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857191 suffix .z20
duplicate parts for zmv 7857191 / suffix .z20 (druid: mx490qf1543, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857191 suffix .z21
duplicate parts for zmv 7857191 / suffix .z21 (druid: mx490qf1543, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857191 suffix .z22
duplicate parts for zmv 7857191 / suffix .z22 (druid: mx490qf1543, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857191 suffix .z23
duplicate parts for zmv 7857191 / suffix .z23 (druid: mx490qf1543, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857191 suffix .z24
duplicate parts for zmv 7857191 / suffix .z24 (druid: mx490qf1543, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857191 suffix .z25
duplicate parts for zmv 7857191 / suffix .z25 (druid: mx490qf1543, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857191 suffix .z26
duplicate parts for zmv 7857191 / suffix .z26 (druid: mx490qf1543, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857191 suffix .z27
duplicate parts for zmv 7857191 / suffix .z27 (druid: mx490qf1543, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857191 suffix .z28
duplicate parts for zmv 7857191 / suffix .z28 (druid: mx490qf1543, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857191 suffix .z29
duplicate parts for zmv 7857191 / suffix .z29 (druid: mx490qf1543, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857191 suffix .z30
duplicate parts for zmv 7857191 / suffix .z30 (druid: mx490qf1543, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857191 suffix .z31
duplicate parts for zmv 7857191 / suffix .z31 (druid: mx490qf1543, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857191 suffix .z32
duplicate parts for zmv 7857191 / suffix .z32 (druid: mx490qf1543, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857191 suffix .z33
duplicate parts for zmv 7857191 / suffix .z33 (druid: mx490qf1543, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857191 suffix .z34
duplicate parts for zmv 7857191 / suffix .z34 (druid: mx490qf1543, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857191 suffix .z35
duplicate parts for zmv 7857191 / suffix .z35 (druid: mx490qf1543, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857191 suffix .z36
duplicate parts for zmv 7857191 / suffix .z36 (druid: mx490qf1543, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857191 suffix .z37
duplicate parts for zmv 7857191 / suffix .z37 (druid: mx490qf1543, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857191 suffix .z38
duplicate parts for zmv 7857191 / suffix .z38 (druid: mx490qf1543, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857191 suffix .z39
duplicate parts for zmv 7857191 / suffix .z39 (druid: mx490qf1543, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857191 suffix .z40
duplicate parts for zmv 7857191 / suffix .z40 (druid: mx490qf1543, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857191 suffix .z41
duplicate parts for zmv 7857191 / suffix .z41 (druid: mx490qf1543, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857191 suffix .z42
duplicate parts for zmv 7857191 / suffix .z42 (druid: mx490qf1543, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857191 suffix .z43
duplicate parts for zmv 7857191 / suffix .z43 (druid: mx490qf1543, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857191 suffix .z44
duplicate parts for zmv 7857191 / suffix .z44 (druid: mx490qf1543, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857191 suffix .z45
duplicate parts for zmv 7857191 / suffix .z45 (druid: mx490qf1543, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857191 suffix .z46
duplicate parts for zmv 7857191 / suffix .z46 (druid: mx490qf1543, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857191 suffix .z47
duplicate parts for zmv 7857191 / suffix .z47 (druid: mx490qf1543, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857191 suffix .z48
duplicate parts for zmv 7857191 / suffix .z48 (druid: mx490qf1543, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857191 suffix .z49
duplicate parts for zmv 7857191 / suffix .z49 (druid: mx490qf1543, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857191 suffix .z50
duplicate parts for zmv 7857191 / suffix .z50 (druid: mx490qf1543, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857191 suffix .z51
duplicate parts for zmv 7857191 / suffix .z51 (druid: mx490qf1543, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857191 suffix .z52
duplicate parts for zmv 7857191 / suffix .z52 (druid: mx490qf1543, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857191 suffix .z53
duplicate parts for zmv 7857191 / suffix .z53 (druid: mx490qf1543, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857191 suffix .z54
duplicate parts for zmv 7857191 / suffix .z54 (druid: mx490qf1543, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857191 suffix .z55
duplicate parts for zmv 7857191 / suffix .z55 (druid: mx490qf1543, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857191 suffix .z56
duplicate parts for zmv 7857191 / suffix .z56 (druid: mx490qf1543, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857191 suffix .zip
duplicate parts for zmv 7857191 / suffix .zip (druid: mx490qf1543, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857203 suffix .z01
duplicate parts for zmv 7857203 / suffix .z01 (druid: tg827dt7345, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857203 suffix .z02
duplicate parts for zmv 7857203 / suffix .z02 (druid: tg827dt7345, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857203 suffix .z03
duplicate parts for zmv 7857203 / suffix .z03 (druid: tg827dt7345, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857203 suffix .z04
duplicate parts for zmv 7857203 / suffix .z04 (druid: tg827dt7345, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857203 suffix .z05
duplicate parts for zmv 7857203 / suffix .z05 (druid: tg827dt7345, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857203 suffix .z06
duplicate parts for zmv 7857203 / suffix .z06 (druid: tg827dt7345, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857203 suffix .z07
duplicate parts for zmv 7857203 / suffix .z07 (druid: tg827dt7345, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857203 suffix .z08
duplicate parts for zmv 7857203 / suffix .z08 (druid: tg827dt7345, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857203 suffix .z09
duplicate parts for zmv 7857203 / suffix .z09 (druid: tg827dt7345, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857203 suffix .z10
duplicate parts for zmv 7857203 / suffix .z10 (druid: tg827dt7345, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857203 suffix .z11
duplicate parts for zmv 7857203 / suffix .z11 (druid: tg827dt7345, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857203 suffix .z12
duplicate parts for zmv 7857203 / suffix .z12 (druid: tg827dt7345, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857203 suffix .z13
duplicate parts for zmv 7857203 / suffix .z13 (druid: tg827dt7345, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857203 suffix .z14
duplicate parts for zmv 7857203 / suffix .z14 (druid: tg827dt7345, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857203 suffix .z15
duplicate parts for zmv 7857203 / suffix .z15 (druid: tg827dt7345, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857203 suffix .z16
duplicate parts for zmv 7857203 / suffix .z16 (druid: tg827dt7345, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857203 suffix .z17
duplicate parts for zmv 7857203 / suffix .z17 (druid: tg827dt7345, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857203 suffix .z18
duplicate parts for zmv 7857203 / suffix .z18 (druid: tg827dt7345, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857203 suffix .z19
duplicate parts for zmv 7857203 / suffix .z19 (druid: tg827dt7345, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857203 suffix .z20
duplicate parts for zmv 7857203 / suffix .z20 (druid: tg827dt7345, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857203 suffix .z21
duplicate parts for zmv 7857203 / suffix .z21 (druid: tg827dt7345, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857203 suffix .z22
duplicate parts for zmv 7857203 / suffix .z22 (druid: tg827dt7345, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857203 suffix .z23
duplicate parts for zmv 7857203 / suffix .z23 (druid: tg827dt7345, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857203 suffix .z24
duplicate parts for zmv 7857203 / suffix .z24 (druid: tg827dt7345, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857203 suffix .z25
duplicate parts for zmv 7857203 / suffix .z25 (druid: tg827dt7345, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857203 suffix .z26
duplicate parts for zmv 7857203 / suffix .z26 (druid: tg827dt7345, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857203 suffix .z27
duplicate parts for zmv 7857203 / suffix .z27 (druid: tg827dt7345, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857203 suffix .z28
duplicate parts for zmv 7857203 / suffix .z28 (druid: tg827dt7345, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857203 suffix .z29
duplicate parts for zmv 7857203 / suffix .z29 (druid: tg827dt7345, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857203 suffix .z30
duplicate parts for zmv 7857203 / suffix .z30 (druid: tg827dt7345, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857203 suffix .z31
duplicate parts for zmv 7857203 / suffix .z31 (druid: tg827dt7345, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857203 suffix .z32
duplicate parts for zmv 7857203 / suffix .z32 (druid: tg827dt7345, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857203 suffix .z33
duplicate parts for zmv 7857203 / suffix .z33 (druid: tg827dt7345, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857203 suffix .z34
duplicate parts for zmv 7857203 / suffix .z34 (druid: tg827dt7345, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857203 suffix .z35
duplicate parts for zmv 7857203 / suffix .z35 (druid: tg827dt7345, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857203 suffix .z36
duplicate parts for zmv 7857203 / suffix .z36 (druid: tg827dt7345, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857203 suffix .z37
duplicate parts for zmv 7857203 / suffix .z37 (druid: tg827dt7345, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857203 suffix .z38
duplicate parts for zmv 7857203 / suffix .z38 (druid: tg827dt7345, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857203 suffix .z39
duplicate parts for zmv 7857203 / suffix .z39 (druid: tg827dt7345, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857203 suffix .z40
duplicate parts for zmv 7857203 / suffix .z40 (druid: tg827dt7345, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857203 suffix .z41
duplicate parts for zmv 7857203 / suffix .z41 (druid: tg827dt7345, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857203 suffix .z42
duplicate parts for zmv 7857203 / suffix .z42 (druid: tg827dt7345, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857203 suffix .z43
duplicate parts for zmv 7857203 / suffix .z43 (druid: tg827dt7345, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857203 suffix .z44
duplicate parts for zmv 7857203 / suffix .z44 (druid: tg827dt7345, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857203 suffix .z47
duplicate parts for zmv 7857203 / suffix .z47 (druid: tg827dt7345, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857203 suffix .z48
duplicate parts for zmv 7857203 / suffix .z48 (druid: tg827dt7345, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857203 suffix .z49
duplicate parts for zmv 7857203 / suffix .z49 (druid: tg827dt7345, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857203 suffix .z50
duplicate parts for zmv 7857203 / suffix .z50 (druid: tg827dt7345, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857203 suffix .z51
duplicate parts for zmv 7857203 / suffix .z51 (druid: tg827dt7345, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857203 suffix .z52
duplicate parts for zmv 7857203 / suffix .z52 (druid: tg827dt7345, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857203 suffix .z53
duplicate parts for zmv 7857203 / suffix .z53 (druid: tg827dt7345, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857203 suffix .z54
duplicate parts for zmv 7857203 / suffix .z54 (druid: tg827dt7345, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857203 suffix .z55
duplicate parts for zmv 7857203 / suffix .z55 (druid: tg827dt7345, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857203 suffix .z56
duplicate parts for zmv 7857203 / suffix .z56 (druid: tg827dt7345, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857203 suffix .z57
duplicate parts for zmv 7857203 / suffix .z57 (druid: tg827dt7345, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857203 suffix .z58
duplicate parts for zmv 7857203 / suffix .z58 (druid: tg827dt7345, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857203 suffix .z59
duplicate parts for zmv 7857203 / suffix .z59 (druid: tg827dt7345, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857203 suffix .z60
duplicate parts for zmv 7857203 / suffix .z60 (druid: tg827dt7345, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857203 suffix .z61
duplicate parts for zmv 7857203 / suffix .z61 (druid: tg827dt7345, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857203 suffix .z62
duplicate parts for zmv 7857203 / suffix .z62 (druid: tg827dt7345, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857203 suffix .z63
duplicate parts for zmv 7857203 / suffix .z63 (druid: tg827dt7345, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857203 suffix .z64
duplicate parts for zmv 7857203 / suffix .z64 (druid: tg827dt7345, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857203 suffix .z65
duplicate parts for zmv 7857203 / suffix .z65 (druid: tg827dt7345, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857203 suffix .z66
duplicate parts for zmv 7857203 / suffix .z66 (druid: tg827dt7345, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857203 suffix .z67
duplicate parts for zmv 7857203 / suffix .z67 (druid: tg827dt7345, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857203 suffix .zip
duplicate parts for zmv 7857203 / suffix .zip (druid: tg827dt7345, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857231 suffix .z138
duplicate parts for zmv 7857231 / suffix .z138 (druid: jf301dx7536, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857231 suffix .z139
duplicate parts for zmv 7857231 / suffix .z139 (druid: jf301dx7536, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857231 suffix .z140
duplicate parts for zmv 7857231 / suffix .z140 (druid: jf301dx7536, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857231 suffix .z141
duplicate parts for zmv 7857231 / suffix .z141 (druid: jf301dx7536, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857231 suffix .z142
duplicate parts for zmv 7857231 / suffix .z142 (druid: jf301dx7536, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857231 suffix .z143
duplicate parts for zmv 7857231 / suffix .z143 (druid: jf301dx7536, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857231 suffix .z144
duplicate parts for zmv 7857231 / suffix .z144 (druid: jf301dx7536, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857231 suffix .z145
duplicate parts for zmv 7857231 / suffix .z145 (druid: jf301dx7536, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857231 suffix .z146
duplicate parts for zmv 7857231 / suffix .z146 (druid: jf301dx7536, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857231 suffix .z147
duplicate parts for zmv 7857231 / suffix .z147 (druid: jf301dx7536, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857231 suffix .z148
duplicate parts for zmv 7857231 / suffix .z148 (druid: jf301dx7536, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857231 suffix .z149
duplicate parts for zmv 7857231 / suffix .z149 (druid: jf301dx7536, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857231 suffix .z150
duplicate parts for zmv 7857231 / suffix .z150 (druid: jf301dx7536, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857231 suffix .z151
duplicate parts for zmv 7857231 / suffix .z151 (druid: jf301dx7536, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857231 suffix .z152
duplicate parts for zmv 7857231 / suffix .z152 (druid: jf301dx7536, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857231 suffix .z153
duplicate parts for zmv 7857231 / suffix .z153 (druid: jf301dx7536, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857231 suffix .z154
duplicate parts for zmv 7857231 / suffix .z154 (druid: jf301dx7536, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857231 suffix .z155
duplicate parts for zmv 7857231 / suffix .z155 (druid: jf301dx7536, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857231 suffix .z156
duplicate parts for zmv 7857231 / suffix .z156 (druid: jf301dx7536, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857231 suffix .z157
duplicate parts for zmv 7857231 / suffix .z157 (druid: jf301dx7536, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857231 suffix .z158
duplicate parts for zmv 7857231 / suffix .z158 (druid: jf301dx7536, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857231 suffix .z159
duplicate parts for zmv 7857231 / suffix .z159 (druid: jf301dx7536, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857231 suffix .z160
duplicate parts for zmv 7857231 / suffix .z160 (druid: jf301dx7536, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857231 suffix .z161
duplicate parts for zmv 7857231 / suffix .z161 (druid: jf301dx7536, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857231 suffix .z162
duplicate parts for zmv 7857231 / suffix .z162 (druid: jf301dx7536, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857231 suffix .z163
duplicate parts for zmv 7857231 / suffix .z163 (druid: jf301dx7536, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857231 suffix .z164
duplicate parts for zmv 7857231 / suffix .z164 (druid: jf301dx7536, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857231 suffix .z165
duplicate parts for zmv 7857231 / suffix .z165 (druid: jf301dx7536, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857231 suffix .z166
duplicate parts for zmv 7857231 / suffix .z166 (druid: jf301dx7536, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857231 suffix .z167
duplicate parts for zmv 7857231 / suffix .z167 (druid: jf301dx7536, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857231 suffix .z168
duplicate parts for zmv 7857231 / suffix .z168 (druid: jf301dx7536, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857231 suffix .z169
duplicate parts for zmv 7857231 / suffix .z169 (druid: jf301dx7536, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857231 suffix .z170
duplicate parts for zmv 7857231 / suffix .z170 (druid: jf301dx7536, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857231 suffix .z171
duplicate parts for zmv 7857231 / suffix .z171 (druid: jf301dx7536, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857231 suffix .z172
duplicate parts for zmv 7857231 / suffix .z172 (druid: jf301dx7536, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857231 suffix .z173
duplicate parts for zmv 7857231 / suffix .z173 (druid: jf301dx7536, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857231 suffix .z174
duplicate parts for zmv 7857231 / suffix .z174 (druid: jf301dx7536, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857231 suffix .z175
duplicate parts for zmv 7857231 / suffix .z175 (druid: jf301dx7536, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857231 suffix .z176
duplicate parts for zmv 7857231 / suffix .z176 (druid: jf301dx7536, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857231 suffix .z177
duplicate parts for zmv 7857231 / suffix .z177 (druid: jf301dx7536, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857231 suffix .z178
duplicate parts for zmv 7857231 / suffix .z178 (druid: jf301dx7536, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857231 suffix .z179
duplicate parts for zmv 7857231 / suffix .z179 (druid: jf301dx7536, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857231 suffix .z180
duplicate parts for zmv 7857231 / suffix .z180 (druid: jf301dx7536, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857231 suffix .z181
duplicate parts for zmv 7857231 / suffix .z181 (druid: jf301dx7536, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857231 suffix .z182
duplicate parts for zmv 7857231 / suffix .z182 (druid: jf301dx7536, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857231 suffix .z183
duplicate parts for zmv 7857231 / suffix .z183 (druid: jf301dx7536, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857231 suffix .z184
duplicate parts for zmv 7857231 / suffix .z184 (druid: jf301dx7536, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857231 suffix .z185
duplicate parts for zmv 7857231 / suffix .z185 (druid: jf301dx7536, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857231 suffix .z190
duplicate parts for zmv 7857231 / suffix .z190 (druid: jf301dx7536, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857231 suffix .z191
duplicate parts for zmv 7857231 / suffix .z191 (druid: jf301dx7536, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857231 suffix .z192
duplicate parts for zmv 7857231 / suffix .z192 (druid: jf301dx7536, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857231 suffix .z193
duplicate parts for zmv 7857231 / suffix .z193 (druid: jf301dx7536, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857231 suffix .z194
duplicate parts for zmv 7857231 / suffix .z194 (druid: jf301dx7536, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857231 suffix .z195
duplicate parts for zmv 7857231 / suffix .z195 (druid: jf301dx7536, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857231 suffix .z196
duplicate parts for zmv 7857231 / suffix .z196 (druid: jf301dx7536, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857231 suffix .z197
duplicate parts for zmv 7857231 / suffix .z197 (druid: jf301dx7536, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857231 suffix .z198
duplicate parts for zmv 7857231 / suffix .z198 (druid: jf301dx7536, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857231 suffix .z199
duplicate parts for zmv 7857231 / suffix .z199 (druid: jf301dx7536, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857231 suffix .z200
duplicate parts for zmv 7857231 / suffix .z200 (druid: jf301dx7536, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857231 suffix .z201
duplicate parts for zmv 7857231 / suffix .z201 (druid: jf301dx7536, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857231 suffix .z202
duplicate parts for zmv 7857231 / suffix .z202 (druid: jf301dx7536, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857231 suffix .z203
duplicate parts for zmv 7857231 / suffix .z203 (druid: jf301dx7536, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857231 suffix .z204
duplicate parts for zmv 7857231 / suffix .z204 (druid: jf301dx7536, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857231 suffix .z205
duplicate parts for zmv 7857231 / suffix .z205 (druid: jf301dx7536, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857231 suffix .z206
duplicate parts for zmv 7857231 / suffix .z206 (druid: jf301dx7536, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857231 suffix .z207
duplicate parts for zmv 7857231 / suffix .z207 (druid: jf301dx7536, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857231 suffix .z208
duplicate parts for zmv 7857231 / suffix .z208 (druid: jf301dx7536, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857231 suffix .z209
duplicate parts for zmv 7857231 / suffix .z209 (druid: jf301dx7536, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857231 suffix .z218
duplicate parts for zmv 7857231 / suffix .z218 (druid: jf301dx7536, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857231 suffix .z219
duplicate parts for zmv 7857231 / suffix .z219 (druid: jf301dx7536, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857231 suffix .z220
duplicate parts for zmv 7857231 / suffix .z220 (druid: jf301dx7536, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857231 suffix .z221
duplicate parts for zmv 7857231 / suffix .z221 (druid: jf301dx7536, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857231 suffix .z222
duplicate parts for zmv 7857231 / suffix .z222 (druid: jf301dx7536, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857231 suffix .z223
duplicate parts for zmv 7857231 / suffix .z223 (druid: jf301dx7536, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857231 suffix .z224
duplicate parts for zmv 7857231 / suffix .z224 (druid: jf301dx7536, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857231 suffix .z225
duplicate parts for zmv 7857231 / suffix .z225 (druid: jf301dx7536, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857231 suffix .z226
duplicate parts for zmv 7857231 / suffix .z226 (druid: jf301dx7536, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857231 suffix .z227
duplicate parts for zmv 7857231 / suffix .z227 (druid: jf301dx7536, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857231 suffix .z228
duplicate parts for zmv 7857231 / suffix .z228 (druid: jf301dx7536, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857231 suffix .z234
duplicate parts for zmv 7857231 / suffix .z234 (druid: jf301dx7536, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857231 suffix .z235
duplicate parts for zmv 7857231 / suffix .z235 (druid: jf301dx7536, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857231 suffix .z243
duplicate parts for zmv 7857231 / suffix .z243 (druid: jf301dx7536, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857231 suffix .z244
duplicate parts for zmv 7857231 / suffix .z244 (druid: jf301dx7536, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857231 suffix .z245
duplicate parts for zmv 7857231 / suffix .z245 (druid: jf301dx7536, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857231 suffix .z246
duplicate parts for zmv 7857231 / suffix .z246 (druid: jf301dx7536, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857231 suffix .z247
duplicate parts for zmv 7857231 / suffix .z247 (druid: jf301dx7536, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857231 suffix .z248
duplicate parts for zmv 7857231 / suffix .z248 (druid: jf301dx7536, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857231 suffix .z249
duplicate parts for zmv 7857231 / suffix .z249 (druid: jf301dx7536, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857231 suffix .z250
duplicate parts for zmv 7857231 / suffix .z250 (druid: jf301dx7536, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857231 suffix .z251
duplicate parts for zmv 7857231 / suffix .z251 (druid: jf301dx7536, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857231 suffix .z265
duplicate parts for zmv 7857231 / suffix .z265 (druid: jf301dx7536, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857231 suffix .z266
duplicate parts for zmv 7857231 / suffix .z266 (druid: jf301dx7536, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857231 suffix .z267
duplicate parts for zmv 7857231 / suffix .z267 (druid: jf301dx7536, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857231 suffix .z268
duplicate parts for zmv 7857231 / suffix .z268 (druid: jf301dx7536, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857231 suffix .z269
duplicate parts for zmv 7857231 / suffix .z269 (druid: jf301dx7536, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857231 suffix .z270
duplicate parts for zmv 7857231 / suffix .z270 (druid: jf301dx7536, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857231 suffix .z271
duplicate parts for zmv 7857231 / suffix .z271 (druid: jf301dx7536, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857231 suffix .z272
duplicate parts for zmv 7857231 / suffix .z272 (druid: jf301dx7536, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 7857231 suffix .z273
duplicate parts for zmv 7857231 / suffix .z273 (druid: jf301dx7536, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8217233 suffix .zip
duplicate parts for zmv 8217233 / suffix .zip (druid: gr404gd5445, version 2, endpoint: aws_s3_west_2) disagree on md5, skipping
analyzing zmv 8311461 suffix .zip
duplicate parts for zmv 8311461 / suffix .zip (druid: mg599hw5271, version 4, endpoint: aws_s3_west_2) disagree on md5, skipping
analyzing zmv 8621036 suffix .z10
duplicate parts for zmv 8621036 / suffix .z10 (druid: qw012qy2533, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8621036 suffix .z14
duplicate parts for zmv 8621036 / suffix .z14 (druid: qw012qy2533, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8621036 suffix .z16
duplicate parts for zmv 8621036 / suffix .z16 (druid: qw012qy2533, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8621036 suffix .z17
duplicate parts for zmv 8621036 / suffix .z17 (druid: qw012qy2533, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8621036 suffix .z18
duplicate parts for zmv 8621036 / suffix .z18 (druid: qw012qy2533, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8621036 suffix .z19
duplicate parts for zmv 8621036 / suffix .z19 (druid: qw012qy2533, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8621036 suffix .z20
duplicate parts for zmv 8621036 / suffix .z20 (druid: qw012qy2533, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8621036 suffix .z21
duplicate parts for zmv 8621036 / suffix .z21 (druid: qw012qy2533, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8621036 suffix .z22
duplicate parts for zmv 8621036 / suffix .z22 (druid: qw012qy2533, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8621036 suffix .z23
duplicate parts for zmv 8621036 / suffix .z23 (druid: qw012qy2533, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8621036 suffix .z24
duplicate parts for zmv 8621036 / suffix .z24 (druid: qw012qy2533, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8621036 suffix .z25
duplicate parts for zmv 8621036 / suffix .z25 (druid: qw012qy2533, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8621036 suffix .z26
duplicate parts for zmv 8621036 / suffix .z26 (druid: qw012qy2533, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8621036 suffix .z27
duplicate parts for zmv 8621036 / suffix .z27 (druid: qw012qy2533, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8621036 suffix .z28
duplicate parts for zmv 8621036 / suffix .z28 (druid: qw012qy2533, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8621036 suffix .z29
duplicate parts for zmv 8621036 / suffix .z29 (druid: qw012qy2533, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8621036 suffix .z30
duplicate parts for zmv 8621036 / suffix .z30 (druid: qw012qy2533, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8621036 suffix .z31
duplicate parts for zmv 8621036 / suffix .z31 (druid: qw012qy2533, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8621036 suffix .z32
duplicate parts for zmv 8621036 / suffix .z32 (druid: qw012qy2533, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8621036 suffix .z33
duplicate parts for zmv 8621036 / suffix .z33 (druid: qw012qy2533, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8621036 suffix .z34
duplicate parts for zmv 8621036 / suffix .z34 (druid: qw012qy2533, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8621036 suffix .z35
duplicate parts for zmv 8621036 / suffix .z35 (druid: qw012qy2533, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8621036 suffix .z36
duplicate parts for zmv 8621036 / suffix .z36 (druid: qw012qy2533, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8621036 suffix .z37
duplicate parts for zmv 8621036 / suffix .z37 (druid: qw012qy2533, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8621036 suffix .z38
duplicate parts for zmv 8621036 / suffix .z38 (druid: qw012qy2533, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8621036 suffix .z39
duplicate parts for zmv 8621036 / suffix .z39 (druid: qw012qy2533, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8621036 suffix .z40
duplicate parts for zmv 8621036 / suffix .z40 (druid: qw012qy2533, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8621036 suffix .z41
duplicate parts for zmv 8621036 / suffix .z41 (druid: qw012qy2533, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8621036 suffix .z42
duplicate parts for zmv 8621036 / suffix .z42 (druid: qw012qy2533, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8621036 suffix .z43
duplicate parts for zmv 8621036 / suffix .z43 (druid: qw012qy2533, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8621036 suffix .z44
duplicate parts for zmv 8621036 / suffix .z44 (druid: qw012qy2533, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8621036 suffix .z45
duplicate parts for zmv 8621036 / suffix .z45 (druid: qw012qy2533, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8621036 suffix .z46
duplicate parts for zmv 8621036 / suffix .z46 (druid: qw012qy2533, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8621036 suffix .z47
duplicate parts for zmv 8621036 / suffix .z47 (druid: qw012qy2533, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8621129 suffix .z14
duplicate parts for zmv 8621129 / suffix .z14 (druid: sr325xz9271, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8621129 suffix .z15
duplicate parts for zmv 8621129 / suffix .z15 (druid: sr325xz9271, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8621129 suffix .z16
duplicate parts for zmv 8621129 / suffix .z16 (druid: sr325xz9271, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8621129 suffix .z17
duplicate parts for zmv 8621129 / suffix .z17 (druid: sr325xz9271, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8621129 suffix .z18
duplicate parts for zmv 8621129 / suffix .z18 (druid: sr325xz9271, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8621129 suffix .z19
duplicate parts for zmv 8621129 / suffix .z19 (druid: sr325xz9271, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8621129 suffix .z20
duplicate parts for zmv 8621129 / suffix .z20 (druid: sr325xz9271, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8621129 suffix .z21
duplicate parts for zmv 8621129 / suffix .z21 (druid: sr325xz9271, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8621129 suffix .z22
duplicate parts for zmv 8621129 / suffix .z22 (druid: sr325xz9271, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8621129 suffix .z23
duplicate parts for zmv 8621129 / suffix .z23 (druid: sr325xz9271, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8621129 suffix .z24
duplicate parts for zmv 8621129 / suffix .z24 (druid: sr325xz9271, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8621129 suffix .z25
duplicate parts for zmv 8621129 / suffix .z25 (druid: sr325xz9271, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8621129 suffix .z26
duplicate parts for zmv 8621129 / suffix .z26 (druid: sr325xz9271, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8621129 suffix .z27
duplicate parts for zmv 8621129 / suffix .z27 (druid: sr325xz9271, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8621129 suffix .z28
duplicate parts for zmv 8621129 / suffix .z28 (druid: sr325xz9271, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8621129 suffix .z29
duplicate parts for zmv 8621129 / suffix .z29 (druid: sr325xz9271, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8621129 suffix .z30
duplicate parts for zmv 8621129 / suffix .z30 (druid: sr325xz9271, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8621129 suffix .z31
duplicate parts for zmv 8621129 / suffix .z31 (druid: sr325xz9271, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8621129 suffix .z32
duplicate parts for zmv 8621129 / suffix .z32 (druid: sr325xz9271, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8621129 suffix .z33
duplicate parts for zmv 8621129 / suffix .z33 (druid: sr325xz9271, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8621129 suffix .z34
duplicate parts for zmv 8621129 / suffix .z34 (druid: sr325xz9271, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8621129 suffix .z35
duplicate parts for zmv 8621129 / suffix .z35 (druid: sr325xz9271, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8621129 suffix .z36
duplicate parts for zmv 8621129 / suffix .z36 (druid: sr325xz9271, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8621129 suffix .z37
duplicate parts for zmv 8621129 / suffix .z37 (druid: sr325xz9271, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8621129 suffix .z38
duplicate parts for zmv 8621129 / suffix .z38 (druid: sr325xz9271, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8621129 suffix .z39
duplicate parts for zmv 8621129 / suffix .z39 (druid: sr325xz9271, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8621129 suffix .z40
duplicate parts for zmv 8621129 / suffix .z40 (druid: sr325xz9271, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8621129 suffix .z41
duplicate parts for zmv 8621129 / suffix .z41 (druid: sr325xz9271, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8621129 suffix .z42
duplicate parts for zmv 8621129 / suffix .z42 (druid: sr325xz9271, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8621129 suffix .z43
duplicate parts for zmv 8621129 / suffix .z43 (druid: sr325xz9271, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8644614 suffix .z15
duplicate parts for zmv 8644614 / suffix .z15 (druid: dc156hp0190, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8644614 suffix .z17
duplicate parts for zmv 8644614 / suffix .z17 (druid: dc156hp0190, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8644614 suffix .z18
duplicate parts for zmv 8644614 / suffix .z18 (druid: dc156hp0190, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8644614 suffix .z19
duplicate parts for zmv 8644614 / suffix .z19 (druid: dc156hp0190, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8644614 suffix .z20
duplicate parts for zmv 8644614 / suffix .z20 (druid: dc156hp0190, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8644614 suffix .z21
duplicate parts for zmv 8644614 / suffix .z21 (druid: dc156hp0190, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8644614 suffix .z22
duplicate parts for zmv 8644614 / suffix .z22 (druid: dc156hp0190, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8644614 suffix .z23
duplicate parts for zmv 8644614 / suffix .z23 (druid: dc156hp0190, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8644614 suffix .z24
duplicate parts for zmv 8644614 / suffix .z24 (druid: dc156hp0190, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8644620 suffix .z35
duplicate parts for zmv 8644620 / suffix .z35 (druid: km388vz4371, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8644620 suffix .z36
duplicate parts for zmv 8644620 / suffix .z36 (druid: km388vz4371, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8644620 suffix .z37
duplicate parts for zmv 8644620 / suffix .z37 (druid: km388vz4371, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8644620 suffix .z38
duplicate parts for zmv 8644620 / suffix .z38 (druid: km388vz4371, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8644620 suffix .z39
duplicate parts for zmv 8644620 / suffix .z39 (druid: km388vz4371, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8644620 suffix .z40
duplicate parts for zmv 8644620 / suffix .z40 (druid: km388vz4371, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8644620 suffix .z41
duplicate parts for zmv 8644620 / suffix .z41 (druid: km388vz4371, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8644620 suffix .z42
duplicate parts for zmv 8644620 / suffix .z42 (druid: km388vz4371, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8644620 suffix .z43
duplicate parts for zmv 8644620 / suffix .z43 (druid: km388vz4371, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8644620 suffix .z44
duplicate parts for zmv 8644620 / suffix .z44 (druid: km388vz4371, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8644620 suffix .z45
duplicate parts for zmv 8644620 / suffix .z45 (druid: km388vz4371, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8644620 suffix .z46
duplicate parts for zmv 8644620 / suffix .z46 (druid: km388vz4371, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8644620 suffix .z47
duplicate parts for zmv 8644620 / suffix .z47 (druid: km388vz4371, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8644620 suffix .z48
duplicate parts for zmv 8644620 / suffix .z48 (druid: km388vz4371, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8644620 suffix .z49
duplicate parts for zmv 8644620 / suffix .z49 (druid: km388vz4371, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8644620 suffix .z50
duplicate parts for zmv 8644620 / suffix .z50 (druid: km388vz4371, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8644620 suffix .z51
duplicate parts for zmv 8644620 / suffix .z51 (druid: km388vz4371, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8644620 suffix .z52
duplicate parts for zmv 8644620 / suffix .z52 (druid: km388vz4371, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8644620 suffix .z53
duplicate parts for zmv 8644620 / suffix .z53 (druid: km388vz4371, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8644620 suffix .z54
duplicate parts for zmv 8644620 / suffix .z54 (druid: km388vz4371, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8644620 suffix .z55
duplicate parts for zmv 8644620 / suffix .z55 (druid: km388vz4371, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8644620 suffix .z56
duplicate parts for zmv 8644620 / suffix .z56 (druid: km388vz4371, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8644620 suffix .z57
duplicate parts for zmv 8644620 / suffix .z57 (druid: km388vz4371, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8644620 suffix .z58
duplicate parts for zmv 8644620 / suffix .z58 (druid: km388vz4371, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8644620 suffix .z59
duplicate parts for zmv 8644620 / suffix .z59 (druid: km388vz4371, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8644620 suffix .z60
duplicate parts for zmv 8644620 / suffix .z60 (druid: km388vz4371, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8644620 suffix .z61
duplicate parts for zmv 8644620 / suffix .z61 (druid: km388vz4371, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8644620 suffix .z62
duplicate parts for zmv 8644620 / suffix .z62 (druid: km388vz4371, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8644620 suffix .z63
duplicate parts for zmv 8644620 / suffix .z63 (druid: km388vz4371, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8644620 suffix .z64
duplicate parts for zmv 8644620 / suffix .z64 (druid: km388vz4371, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8644620 suffix .z65
duplicate parts for zmv 8644620 / suffix .z65 (druid: km388vz4371, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8644620 suffix .z66
duplicate parts for zmv 8644620 / suffix .z66 (druid: km388vz4371, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8644620 suffix .z67
duplicate parts for zmv 8644620 / suffix .z67 (druid: km388vz4371, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8644620 suffix .z68
duplicate parts for zmv 8644620 / suffix .z68 (druid: km388vz4371, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8644620 suffix .z69
duplicate parts for zmv 8644620 / suffix .z69 (druid: km388vz4371, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8644620 suffix .z70
duplicate parts for zmv 8644620 / suffix .z70 (druid: km388vz4371, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8644620 suffix .z71
duplicate parts for zmv 8644620 / suffix .z71 (druid: km388vz4371, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8644626 suffix .z17
duplicate parts for zmv 8644626 / suffix .z17 (druid: kp222tm1554, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8644626 suffix .z18
duplicate parts for zmv 8644626 / suffix .z18 (druid: kp222tm1554, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8644626 suffix .z19
duplicate parts for zmv 8644626 / suffix .z19 (druid: kp222tm1554, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8644626 suffix .z20
duplicate parts for zmv 8644626 / suffix .z20 (druid: kp222tm1554, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8644626 suffix .z21
duplicate parts for zmv 8644626 / suffix .z21 (druid: kp222tm1554, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8644626 suffix .z22
duplicate parts for zmv 8644626 / suffix .z22 (druid: kp222tm1554, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8644626 suffix .z23
duplicate parts for zmv 8644626 / suffix .z23 (druid: kp222tm1554, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8644626 suffix .z24
duplicate parts for zmv 8644626 / suffix .z24 (druid: kp222tm1554, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8644626 suffix .z25
duplicate parts for zmv 8644626 / suffix .z25 (druid: kp222tm1554, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8644626 suffix .z26
duplicate parts for zmv 8644626 / suffix .z26 (druid: kp222tm1554, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8644626 suffix .z27
duplicate parts for zmv 8644626 / suffix .z27 (druid: kp222tm1554, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8644626 suffix .z28
duplicate parts for zmv 8644626 / suffix .z28 (druid: kp222tm1554, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8644626 suffix .z29
duplicate parts for zmv 8644626 / suffix .z29 (druid: kp222tm1554, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8644626 suffix .z30
duplicate parts for zmv 8644626 / suffix .z30 (druid: kp222tm1554, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8644626 suffix .z31
duplicate parts for zmv 8644626 / suffix .z31 (druid: kp222tm1554, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 8800763 suffix .z32
duplicate parts for zmv 8800763 / suffix .z32 (druid: jc488jb7715, version 1, endpoint: ibm_us_south) disagree on parts_count, skipping
analyzing zmv 8800763 suffix .z33
duplicate parts for zmv 8800763 / suffix .z33 (druid: jc488jb7715, version 1, endpoint: ibm_us_south) disagree on parts_count, skipping
analyzing zmv 8800763 suffix .z34
duplicate parts for zmv 8800763 / suffix .z34 (druid: jc488jb7715, version 1, endpoint: ibm_us_south) disagree on parts_count, skipping
analyzing zmv 8800763 suffix .z35
duplicate parts for zmv 8800763 / suffix .z35 (druid: jc488jb7715, version 1, endpoint: ibm_us_south) disagree on parts_count, skipping
analyzing zmv 8800763 suffix .z36
duplicate parts for zmv 8800763 / suffix .z36 (druid: jc488jb7715, version 1, endpoint: ibm_us_south) disagree on parts_count, skipping
analyzing zmv 8800763 suffix .z37
duplicate parts for zmv 8800763 / suffix .z37 (druid: jc488jb7715, version 1, endpoint: ibm_us_south) disagree on parts_count, skipping
analyzing zmv 8800763 suffix .z38
duplicate parts for zmv 8800763 / suffix .z38 (druid: jc488jb7715, version 1, endpoint: ibm_us_south) disagree on parts_count, skipping
analyzing zmv 8800763 suffix .z39
duplicate parts for zmv 8800763 / suffix .z39 (druid: jc488jb7715, version 1, endpoint: ibm_us_south) disagree on parts_count, skipping
analyzing zmv 8800763 suffix .z40
duplicate parts for zmv 8800763 / suffix .z40 (druid: jc488jb7715, version 1, endpoint: ibm_us_south) disagree on parts_count, skipping
analyzing zmv 8800763 suffix .z41
duplicate parts for zmv 8800763 / suffix .z41 (druid: jc488jb7715, version 1, endpoint: ibm_us_south) disagree on parts_count, skipping
analyzing zmv 8800763 suffix .z42
duplicate parts for zmv 8800763 / suffix .z42 (druid: jc488jb7715, version 1, endpoint: ibm_us_south) disagree on parts_count, skipping
analyzing zmv 8800763 suffix .z43
duplicate parts for zmv 8800763 / suffix .z43 (druid: jc488jb7715, version 1, endpoint: ibm_us_south) disagree on parts_count, skipping
analyzing zmv 8800763 suffix .z44
duplicate parts for zmv 8800763 / suffix .z44 (druid: jc488jb7715, version 1, endpoint: ibm_us_south) disagree on parts_count, skipping
analyzing zmv 8800763 suffix .z45
duplicate parts for zmv 8800763 / suffix .z45 (druid: jc488jb7715, version 1, endpoint: ibm_us_south) disagree on parts_count, skipping
analyzing zmv 8800763 suffix .z46
duplicate parts for zmv 8800763 / suffix .z46 (druid: jc488jb7715, version 1, endpoint: ibm_us_south) disagree on parts_count, skipping
analyzing zmv 8800763 suffix .z47
duplicate parts for zmv 8800763 / suffix .z47 (druid: jc488jb7715, version 1, endpoint: ibm_us_south) disagree on parts_count, skipping
analyzing zmv 8800763 suffix .z48
duplicate parts for zmv 8800763 / suffix .z48 (druid: jc488jb7715, version 1, endpoint: ibm_us_south) disagree on parts_count, skipping
analyzing zmv 8800763 suffix .z49
duplicate parts for zmv 8800763 / suffix .z49 (druid: jc488jb7715, version 1, endpoint: ibm_us_south) disagree on parts_count, skipping
analyzing zmv 8800763 suffix .z50
duplicate parts for zmv 8800763 / suffix .z50 (druid: jc488jb7715, version 1, endpoint: ibm_us_south) disagree on parts_count, skipping
analyzing zmv 8800763 suffix .z51
duplicate parts for zmv 8800763 / suffix .z51 (druid: jc488jb7715, version 1, endpoint: ibm_us_south) disagree on parts_count, skipping
analyzing zmv 8800763 suffix .z52
duplicate parts for zmv 8800763 / suffix .z52 (druid: jc488jb7715, version 1, endpoint: ibm_us_south) disagree on parts_count, skipping
analyzing zmv 8800763 suffix .z53
duplicate parts for zmv 8800763 / suffix .z53 (druid: jc488jb7715, version 1, endpoint: ibm_us_south) disagree on parts_count, skipping
analyzing zmv 8800763 suffix .z54
duplicate parts for zmv 8800763 / suffix .z54 (druid: jc488jb7715, version 1, endpoint: ibm_us_south) disagree on parts_count, skipping
analyzing zmv 8800763 suffix .z55
duplicate parts for zmv 8800763 / suffix .z55 (druid: jc488jb7715, version 1, endpoint: ibm_us_south) disagree on parts_count, skipping
analyzing zmv 8800763 suffix .z56
duplicate parts for zmv 8800763 / suffix .z56 (druid: jc488jb7715, version 1, endpoint: ibm_us_south) disagree on parts_count, skipping
analyzing zmv 8800763 suffix .z57
duplicate parts for zmv 8800763 / suffix .z57 (druid: jc488jb7715, version 1, endpoint: ibm_us_south) disagree on parts_count, skipping
analyzing zmv 8800763 suffix .z58
duplicate parts for zmv 8800763 / suffix .z58 (druid: jc488jb7715, version 1, endpoint: ibm_us_south) disagree on parts_count, skipping
analyzing zmv 8800763 suffix .z59
duplicate parts for zmv 8800763 / suffix .z59 (druid: jc488jb7715, version 1, endpoint: ibm_us_south) disagree on parts_count, skipping
analyzing zmv 8800763 suffix .z60
duplicate parts for zmv 8800763 / suffix .z60 (druid: jc488jb7715, version 1, endpoint: ibm_us_south) disagree on parts_count, skipping
analyzing zmv 8800763 suffix .z61
duplicate parts for zmv 8800763 / suffix .z61 (druid: jc488jb7715, version 1, endpoint: ibm_us_south) disagree on parts_count, skipping
analyzing zmv 8800763 suffix .z62
duplicate parts for zmv 8800763 / suffix .z62 (druid: jc488jb7715, version 1, endpoint: ibm_us_south) disagree on parts_count, skipping
analyzing zmv 8800763 suffix .z63
duplicate parts for zmv 8800763 / suffix .z63 (druid: jc488jb7715, version 1, endpoint: ibm_us_south) disagree on parts_count, skipping
analyzing zmv 8800763 suffix .z64
duplicate parts for zmv 8800763 / suffix .z64 (druid: jc488jb7715, version 1, endpoint: ibm_us_south) disagree on parts_count, skipping
analyzing zmv 8800763 suffix .z65
duplicate parts for zmv 8800763 / suffix .z65 (druid: jc488jb7715, version 1, endpoint: ibm_us_south) disagree on parts_count, skipping
analyzing zmv 8800763 suffix .z66
duplicate parts for zmv 8800763 / suffix .z66 (druid: jc488jb7715, version 1, endpoint: ibm_us_south) disagree on parts_count, skipping
analyzing zmv 8800763 suffix .z67
duplicate parts for zmv 8800763 / suffix .z67 (druid: jc488jb7715, version 1, endpoint: ibm_us_south) disagree on parts_count, skipping
analyzing zmv 8800763 suffix .z68
duplicate parts for zmv 8800763 / suffix .z68 (druid: jc488jb7715, version 1, endpoint: ibm_us_south) disagree on parts_count, skipping
analyzing zmv 8800763 suffix .z69
duplicate parts for zmv 8800763 / suffix .z69 (druid: jc488jb7715, version 1, endpoint: ibm_us_south) disagree on parts_count, skipping
analyzing zmv 8800763 suffix .z70
duplicate parts for zmv 8800763 / suffix .z70 (druid: jc488jb7715, version 1, endpoint: ibm_us_south) disagree on parts_count, skipping
analyzing zmv 8800763 suffix .zip
duplicate parts for zmv 8800763 / suffix .zip (druid: jc488jb7715, version 1, endpoint: ibm_us_south) disagree on parts_count, skipping
analyzing zmv 8800806 suffix .z37
duplicate parts for zmv 8800806 / suffix .z37 (druid: nk828sc2920, version 1, endpoint: ibm_us_south) disagree on parts_count, skipping
analyzing zmv 8800806 suffix .z38
duplicate parts for zmv 8800806 / suffix .z38 (druid: nk828sc2920, version 1, endpoint: ibm_us_south) disagree on parts_count, skipping
analyzing zmv 8800806 suffix .z39
duplicate parts for zmv 8800806 / suffix .z39 (druid: nk828sc2920, version 1, endpoint: ibm_us_south) disagree on parts_count, skipping
analyzing zmv 8800806 suffix .z40
duplicate parts for zmv 8800806 / suffix .z40 (druid: nk828sc2920, version 1, endpoint: ibm_us_south) disagree on parts_count, skipping
analyzing zmv 8800806 suffix .z41
duplicate parts for zmv 8800806 / suffix .z41 (druid: nk828sc2920, version 1, endpoint: ibm_us_south) disagree on parts_count, skipping
analyzing zmv 8800806 suffix .z42
duplicate parts for zmv 8800806 / suffix .z42 (druid: nk828sc2920, version 1, endpoint: ibm_us_south) disagree on parts_count, skipping
analyzing zmv 8800806 suffix .z43
duplicate parts for zmv 8800806 / suffix .z43 (druid: nk828sc2920, version 1, endpoint: ibm_us_south) disagree on parts_count, skipping
analyzing zmv 8800806 suffix .z44
duplicate parts for zmv 8800806 / suffix .z44 (druid: nk828sc2920, version 1, endpoint: ibm_us_south) disagree on parts_count, skipping
analyzing zmv 8800806 suffix .z45
duplicate parts for zmv 8800806 / suffix .z45 (druid: nk828sc2920, version 1, endpoint: ibm_us_south) disagree on parts_count, skipping
analyzing zmv 8800806 suffix .z46
duplicate parts for zmv 8800806 / suffix .z46 (druid: nk828sc2920, version 1, endpoint: ibm_us_south) disagree on parts_count, skipping
analyzing zmv 8800806 suffix .z47
duplicate parts for zmv 8800806 / suffix .z47 (druid: nk828sc2920, version 1, endpoint: ibm_us_south) disagree on parts_count, skipping
analyzing zmv 8800806 suffix .z48
duplicate parts for zmv 8800806 / suffix .z48 (druid: nk828sc2920, version 1, endpoint: ibm_us_south) disagree on parts_count, skipping
analyzing zmv 8800806 suffix .z49
duplicate parts for zmv 8800806 / suffix .z49 (druid: nk828sc2920, version 1, endpoint: ibm_us_south) disagree on parts_count, skipping
analyzing zmv 8800806 suffix .z50
duplicate parts for zmv 8800806 / suffix .z50 (druid: nk828sc2920, version 1, endpoint: ibm_us_south) disagree on parts_count, skipping
analyzing zmv 8800806 suffix .z51
duplicate parts for zmv 8800806 / suffix .z51 (druid: nk828sc2920, version 1, endpoint: ibm_us_south) disagree on parts_count, skipping
analyzing zmv 8800806 suffix .z52
duplicate parts for zmv 8800806 / suffix .z52 (druid: nk828sc2920, version 1, endpoint: ibm_us_south) disagree on parts_count, skipping
analyzing zmv 8800806 suffix .z53
duplicate parts for zmv 8800806 / suffix .z53 (druid: nk828sc2920, version 1, endpoint: ibm_us_south) disagree on parts_count, skipping
analyzing zmv 8800806 suffix .z54
duplicate parts for zmv 8800806 / suffix .z54 (druid: nk828sc2920, version 1, endpoint: ibm_us_south) disagree on parts_count, skipping
analyzing zmv 8800806 suffix .z55
duplicate parts for zmv 8800806 / suffix .z55 (druid: nk828sc2920, version 1, endpoint: ibm_us_south) disagree on parts_count, skipping
analyzing zmv 8800806 suffix .z56
duplicate parts for zmv 8800806 / suffix .z56 (druid: nk828sc2920, version 1, endpoint: ibm_us_south) disagree on parts_count, skipping
analyzing zmv 8800806 suffix .z57
duplicate parts for zmv 8800806 / suffix .z57 (druid: nk828sc2920, version 1, endpoint: ibm_us_south) disagree on parts_count, skipping
analyzing zmv 8800806 suffix .z58
duplicate parts for zmv 8800806 / suffix .z58 (druid: nk828sc2920, version 1, endpoint: ibm_us_south) disagree on parts_count, skipping
analyzing zmv 8800806 suffix .z59
duplicate parts for zmv 8800806 / suffix .z59 (druid: nk828sc2920, version 1, endpoint: ibm_us_south) disagree on parts_count, skipping
analyzing zmv 8800806 suffix .z60
duplicate parts for zmv 8800806 / suffix .z60 (druid: nk828sc2920, version 1, endpoint: ibm_us_south) disagree on parts_count, skipping
analyzing zmv 8800806 suffix .z61
duplicate parts for zmv 8800806 / suffix .z61 (druid: nk828sc2920, version 1, endpoint: ibm_us_south) disagree on parts_count, skipping
analyzing zmv 8800806 suffix .z62
duplicate parts for zmv 8800806 / suffix .z62 (druid: nk828sc2920, version 1, endpoint: ibm_us_south) disagree on parts_count, skipping
analyzing zmv 8800806 suffix .z63
duplicate parts for zmv 8800806 / suffix .z63 (druid: nk828sc2920, version 1, endpoint: ibm_us_south) disagree on parts_count, skipping
analyzing zmv 8800806 suffix .z64
duplicate parts for zmv 8800806 / suffix .z64 (druid: nk828sc2920, version 1, endpoint: ibm_us_south) disagree on parts_count, skipping
analyzing zmv 8800806 suffix .z65
duplicate parts for zmv 8800806 / suffix .z65 (druid: nk828sc2920, version 1, endpoint: ibm_us_south) disagree on parts_count, skipping
analyzing zmv 8800806 suffix .zip
duplicate parts for zmv 8800806 / suffix .zip (druid: nk828sc2920, version 1, endpoint: ibm_us_south) disagree on parts_count, skipping
analyzing zmv 9067945 suffix .zip
duplicate parts for zmv 9067945 / suffix .zip (druid: jq797jt1037, version 1, endpoint: ibm_us_south) disagree on md5, skipping
analyzing zmv 12387333 suffix .zip
duplicate parts for zmv 12387333 / suffix .zip (druid: fm001pk4253, version 1, endpoint: ibm_us_south) disagree on md5, skipping
analyzing zmv 13296953 suffix .z01
duplicate parts for zmv 13296953 / suffix .z01 (druid: zg502tr5183, version 1, endpoint: ibm_us_south) disagree on parts_count, skipping
analyzing zmv 13296953 suffix .z13
duplicate parts for zmv 13296953 / suffix .z13 (druid: zg502tr5183, version 1, endpoint: ibm_us_south) disagree on parts_count, skipping
analyzing zmv 13296953 suffix .z14
duplicate parts for zmv 13296953 / suffix .z14 (druid: zg502tr5183, version 1, endpoint: ibm_us_south) disagree on parts_count, skipping
analyzing zmv 13296953 suffix .z15
duplicate parts for zmv 13296953 / suffix .z15 (druid: zg502tr5183, version 1, endpoint: ibm_us_south) disagree on parts_count, skipping
analyzing zmv 13296953 suffix .zip
duplicate parts for zmv 13296953 / suffix .zip (druid: zg502tr5183, version 1, endpoint: ibm_us_south) disagree on parts_count, skipping
analyzing zmv 13297719 suffix .z03
duplicate parts for zmv 13297719 / suffix .z03 (druid: zg936mz3511, version 1, endpoint: ibm_us_south) disagree on parts_count, skipping
analyzing zmv 13297719 suffix .z08
duplicate parts for zmv 13297719 / suffix .z08 (druid: zg936mz3511, version 1, endpoint: ibm_us_south) disagree on parts_count, skipping
analyzing zmv 13297719 suffix .zip
duplicate parts for zmv 13297719 / suffix .zip (druid: zg936mz3511, version 1, endpoint: ibm_us_south) disagree on parts_count, skipping
analyzing zmv 13397782 suffix .z01
duplicate parts for zmv 13397782 / suffix .z01 (druid: bk375rd3008, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 13397782 suffix .z03
duplicate parts for zmv 13397782 / suffix .z03 (druid: bk375rd3008, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 13397782 suffix .zip
duplicate parts for zmv 13397782 / suffix .zip (druid: bk375rd3008, version 1, endpoint: aws_s3_west_2) disagree on parts_count, skipping
analyzing zmv 13397783 suffix .z01
duplicate parts for zmv 13397783 / suffix .z01 (druid: bk375rd3008, version 1, endpoint: ibm_us_south) disagree on parts_count, skipping
analyzing zmv 13397783 suffix .z03
duplicate parts for zmv 13397783 / suffix .z03 (druid: bk375rd3008, version 1, endpoint: ibm_us_south) disagree on parts_count, skipping
analyzing zmv 13397783 suffix .zip
duplicate parts for zmv 13397783 / suffix .zip (druid: bk375rd3008, version 1, endpoint: ibm_us_south) disagree on parts_count, skipping
analyzing zmv 13572723 suffix .z01
duplicate parts for zmv 13572723 / suffix .z01 (druid: wz600xp8911, version 1, endpoint: ibm_us_south) disagree on parts_count, skipping
analyzing zmv 13572723 suffix .z02
duplicate parts for zmv 13572723 / suffix .z02 (druid: wz600xp8911, version 1, endpoint: ibm_us_south) disagree on parts_count, skipping
analyzing zmv 13572723 suffix .z03
duplicate parts for zmv 13572723 / suffix .z03 (druid: wz600xp8911, version 1, endpoint: ibm_us_south) disagree on parts_count, skipping
analyzing zmv 13572723 suffix .z04
duplicate parts for zmv 13572723 / suffix .z04 (druid: wz600xp8911, version 1, endpoint: ibm_us_south) disagree on parts_count, skipping
analyzing zmv 13572723 suffix .z05
duplicate parts for zmv 13572723 / suffix .z05 (druid: wz600xp8911, version 1, endpoint: ibm_us_south) disagree on parts_count, skipping
analyzing zmv 13572723 suffix .z06
duplicate parts for zmv 13572723 / suffix .z06 (druid: wz600xp8911, version 1, endpoint: ibm_us_south) disagree on parts_count, skipping
analyzing zmv 13572723 suffix .z07
duplicate parts for zmv 13572723 / suffix .z07 (druid: wz600xp8911, version 1, endpoint: ibm_us_south) disagree on parts_count, skipping
analyzing zmv 13572723 suffix .z08
duplicate parts for zmv 13572723 / suffix .z08 (druid: wz600xp8911, version 1, endpoint: ibm_us_south) disagree on parts_count, skipping
analyzing zmv 13572723 suffix .z09
duplicate parts for zmv 13572723 / suffix .z09 (druid: wz600xp8911, version 1, endpoint: ibm_us_south) disagree on parts_count, skipping
analyzing zmv 13572723 suffix .z10
duplicate parts for zmv 13572723 / suffix .z10 (druid: wz600xp8911, version 1, endpoint: ibm_us_south) disagree on parts_count, skipping
analyzing zmv 13572723 suffix .z11
duplicate parts for zmv 13572723 / suffix .z11 (druid: wz600xp8911, version 1, endpoint: ibm_us_south) disagree on parts_count, skipping
analyzing zmv 13572723 suffix .z12
duplicate parts for zmv 13572723 / suffix .z12 (druid: wz600xp8911, version 1, endpoint: ibm_us_south) disagree on parts_count, skipping
analyzing zmv 13572723 suffix .z13
duplicate parts for zmv 13572723 / suffix .z13 (druid: wz600xp8911, version 1, endpoint: ibm_us_south) disagree on parts_count, skipping
analyzing zmv 13572723 suffix .z14
duplicate parts for zmv 13572723 / suffix .z14 (druid: wz600xp8911, version 1, endpoint: ibm_us_south) disagree on parts_count, skipping
analyzing zmv 13572723 suffix .z15
duplicate parts for zmv 13572723 / suffix .z15 (druid: wz600xp8911, version 1, endpoint: ibm_us_south) disagree on parts_count, skipping
analyzing zmv 13572723 suffix .z16
duplicate parts for zmv 13572723 / suffix .z16 (druid: wz600xp8911, version 1, endpoint: ibm_us_south) disagree on parts_count, skipping
analyzing zmv 13572723 suffix .z17
duplicate parts for zmv 13572723 / suffix .z17 (druid: wz600xp8911, version 1, endpoint: ibm_us_south) disagree on parts_count, skipping
analyzing zmv 13572723 suffix .z18
duplicate parts for zmv 13572723 / suffix .z18 (druid: wz600xp8911, version 1, endpoint: ibm_us_south) disagree on parts_count, skipping
analyzing zmv 13572723 suffix .z19
duplicate parts for zmv 13572723 / suffix .z19 (druid: wz600xp8911, version 1, endpoint: ibm_us_south) disagree on parts_count, skipping
analyzing zmv 13572723 suffix .z20
duplicate parts for zmv 13572723 / suffix .z20 (druid: wz600xp8911, version 1, endpoint: ibm_us_south) disagree on parts_count, skipping
analyzing zmv 13572723 suffix .z21
duplicate parts for zmv 13572723 / suffix .z21 (druid: wz600xp8911, version 1, endpoint: ibm_us_south) disagree on parts_count, skipping
analyzing zmv 13572723 suffix .z22
duplicate parts for zmv 13572723 / suffix .z22 (druid: wz600xp8911, version 1, endpoint: ibm_us_south) disagree on parts_count, skipping
analyzing zmv 13577006 suffix .zip
duplicate parts for zmv 13577006 / suffix .zip (druid: rn821pp7681, version 1, endpoint: aws_s3_west_2) disagree on md5, skipping
analyzing zmv 13577007 suffix .zip
duplicate parts for zmv 13577007 / suffix .zip (druid: rn821pp7681, version 1, endpoint: ibm_us_south) disagree on md5, skipping
analyzing zmv 13622662 suffix .zip
duplicate parts for zmv 13622662 / suffix .zip (druid: bb003ym5750, version 1, endpoint: aws_s3_west_2) disagree on md5, skipping
analyzing zmv 13622663 suffix .zip
duplicate parts for zmv 13622663 / suffix .zip (druid: bb003ym5750, version 1, endpoint: ibm_us_south) disagree on md5, skipping
analyzing zmv 19651474 suffix .z13
duplicate parts for zmv 19651474 / suffix .z13 (druid: zg502tr5183, version 1, endpoint: aws_s3_east_1) disagree on parts_count, skipping
analyzing zmv 19651474 suffix .zip
duplicate parts for zmv 19651474 / suffix .zip (druid: zg502tr5183, version 1, endpoint: aws_s3_east_1) disagree on parts_count, skipping

[mjgiarlo]

I have remediated the ZMV with two unreplicated zip parts by destroying one with a bogus parts_count and setting the other to OK.

[jmartin-sul]

the zmv_dupes.each do |zmv_id, suffix| looping remediation script looks good to me (including the commented out remediation at the end)!

The majority of the "one ok, one unreplicated" ZMV dupes agree on parts_count, size, and md5. That's 4,076 of the 4,569. For all of those, all replicated parts exist.

that's great!

Here are the (493) problematic ones:

interesting (maybe comforting?) that, at first glance, most of the parts mentioned in that list seem to belong to a few very large moab versions?

[mjgiarlo]

After running an earlier, simplified version of the remediation above, here's the most recent analysis:

Number	Description
975 (was 983)	2 OK ZipParts
493 (was 4,569)	1 OK ZipPart, 1 unreplicated ZipPart
348 (was 351)	3 or more ZipParts (combination of various OKs and unreplicateds
0 (was 1)	2 unreplicated ZipParts

The rows above need more logic (around mismatched parts_count/md5/size numbers) in order to be remediated. @jmartin-sul

[mjgiarlo]

I have documented this remediation here: https://github.com/sul-dlss/preservation_catalog/wiki/Investigating-a-druid-with-replication-errors#code-for-de-duplicating-zip-parts-in-the-database-based-on-cloud-state

[mjgiarlo]

After running the full remediation, there are only two affected remaining rows:

ZippedMoabVersion: 13397783, suffixes: ".z03", ".zip"

[mjgiarlo]

I edited the remediation and we're down to zero affected rows.

[mjgiarlo]

@jmartin-sul think we ought to deploy this today to protect against having to do this remediation again?