Unverified list contains 444 gems which are not alpha/pre-release

[heathd]

Hi,

I think you have a bug in this command:

ruby -ne 'gets =~ /-(.*?)\.gem/; puts $_ unless $1 =~ /\w/' \
  unverified.2_or_fewer_checksums.txt

the problem is that \w matches numbers as well as letters. The correct command would be:

ruby -ne 'gets =~ /-(.*?)\.gem/; puts $_ unless $1 =~ /[a-zA-Z]' \
  unverified.2_or_fewer_checksums.txt

This finds 444 gems which are full versions.

PD_officeally-0.1.1.gem 3 1
PD_officeally-0.1.3.gem 3 1
PD_officeally-0.1.5.gem 3 1
PD_x12-1.3.12.gem 3 1
PD_x12-1.3.7.gem 3 1
PD_x12-1.3.9.gem 3 1
RubyApp-0.6.52.gem 3 1
Sly-0.0.1.gem 3 1
aa_associations-0.1.2.gem 3 1
acl_system2-0.2.0.gem 3 1
active_remote-1.3.3.gem 3 1
active_scaffold_vho-3.0.29.gem 3 1
active_scaffold_vho-3.1.6.gem 3 1
acts_as_indexed-0.8.2.gem 3 1
admix-0.16.0.gem 3 1
af-0.3.18.12.gem 3 1
air18n-0.1.56.gem 3 1
aktion_test-0.2.1.gem 3 1
area-0.7.0.gem 3 1
arg0-0.0.1.gem 3 1
asciiart-0.0.5.gem 3 1
attr_unit-1.0.0.gem 3 1
attr_unit-1.0.2.gem 3 1
audite-0.1.0.gem 3 1
audite-0.1.2.gem 3 1
audite-0.1.4.gem 3 1
audite-0.1.6.gem 3 1
audite-0.3.0.gem 3 1
audited_controller-0.0.2.gem 3 1
audited_controller-0.0.4.gem 3 1
audited_controller-0.0.6.gem 3 1
awsome-0.0.25.gem 3 1
awsome-0.0.27.gem 3 1
awsome-0.0.29.gem 3 1
awsome-0.0.31.gem 3 1
awsome-0.0.33.gem 3 1
awsome-0.0.35.gem 3 1
awsome-0.0.37.gem 3 1
awsome-0.0.39.gem 3 1
awsome-0.0.41.gem 3 1
awsome-0.0.43.gem 3 1
awsome-0.0.45.gem 3 1
babysitter-0.0.6.gem 3 1
background_lite-0.4.gem 3 1
bandcamp_api-0.1.0.gem 3 1
bard-0.17.0.gem 3 1
better_receive-0.3.0.gem 3 1
bhf-0.4.30.gem 3 1
blogaze-0.1.0.gem 3 1
bloomin-0.0.1.gem 3 1
boarding_pass-0.1.4.gem 3 1
book_finder_api-0.0.16.gem 3 1
boot_polish-0.0.1.gem 3 1
braai-1.4.0.gem 3 1
brails-0.0.7.gem 3 1
brails-0.1.0.gem 3 1
bulstem-0.0.1.gem 3 1
caff-0.0.1.gem 3 1
can_be-0.3.0.gem 3 1
cane-2.5.1.gem 3 1
cdnio-0.1.0.gem 3 1
chap-0.0.2.gem 3 1
cheddargetter_client_rails-0.2.2.gem 3 1
china_pay-0.0.1.gem 3 1
churn-0.0.27.gem 3 1
cipher_bureau-0.1.0.gem 3 1
ckeditor-4.0.2.gem 3 1
classnamer-3.0.0.gem 3 1
clavem-1.0.1.gem 3 1
clavem-1.0.3.gem 3 1
closure_tree-3.7.1.gem 3 1
clutter-1.2.1.gem 3 1
coast-0.9.9.gem 3 1
coinbase-1.1.1.gem 3 1
collector-0.1.1.gem 3 1
collins_notify-0.0.2.gem 3 1
colony-0.1.32.gem 3 1
compass_radix-0.0.6.gem 3 1
compass_radix-0.0.8.gem 3 1
connection_manager-0.3.8.gem 3 1
contacts_client-0.0.7.gem 3 1
contacts_client-0.0.9.gem 3 1
copiedbook-0.0.1.gem 3 1
copy_by_xumc-0.0.0.gem 3 1
copyscape-0.0.5.gem 3 1
crowdtap_redis_lock-0.1.1.gem 3 1
cul_image_props-0.3.4.gem 3 1
cv4r-0.0.1.gem 3 1
cytoplasm-0.2.5.gem 3 1
daengine-0.4.6.2.gem 3 1
dandelion-0.3.10.gem 3 1
databasedotcom_console-0.0.1.gem 3 1
databasedotcom_console-0.0.3.gem 3 1
databasedotcom_console-0.0.5.gem 3 1
databasedotcom_console-0.0.7.gem 3 1
datatrans-2.4.0.gem 3 1
dbmule-0.1.1.gem 3 1
debugger-1.3.0.gem 3 1
defaulter-0.0.9.gem 3 1
delorean_lang-0.0.34.gem 3 1
dev-1.0.81.gem 3 1
dev-1.0.83.gem 3 1
devcenter-0.0.4.gem 3 1
developer_info-0.0.1.gem 3 1
disqussion-0.0.6.gem 3 1
dot_hash-0.3.1.gem 3 1
dotopts-0.1.1.gem 3 1
double_double-0.1.0.gem 3 1
dragonfly-0.9.13.gem 3 1
dynamic_query-0.4.0.gem 3 1
eader-0.0.4.gem 3 1
easy_shell-0.0.1.gem 3 1
elasticsearch_autocomplete-0.0.3.gem 3 1
em_aws-0.2.2.gem 3 1
enju_trunk_ill-0.1.gem 3 1
entasis-0.4.0.gem 3 1
environment_initializers-0.0.1.gem 3 1
event_bus-0.0.4.gem 3 1
event_cal-1.2.1.gem 3 1
event_cal-1.2.3.gem 3 1
event_cal-1.2.gem 3 1
fakeapi-0.0.2.gem 3 1
fb_graph-2.6.4.gem 3 1
filbert-0.0.5.gem 3 1
file_sort-0.0.2.gem 3 1
filenc-0.0.3.gem 3 1
finfast-0.0.4.gem 3 1
fixed_length_encoder-1.2.1.gem 3 1
flickrup-1.1.0.gem 3 1
flightplan-3.1.2.gem 3 1
flightplan-3.1.9.gem 3 1
flunk-0.0.9.gem 3 1
formatted_form-2.1.0.gem 3 1
fromation-0.0.2.gem 3 1
ga_events-0.1.0.gem 3 1
gametel-0.5.5.gem 3 1
gates_of_moria-0.0.5.gem 3 1
gates_of_moria-0.0.7.gem 3 1
gdk3-1.2.1.gem 3 1
gdk_pixbuf2-1.2.1.gem 3 1
geekier_factory-0.0.1.gem 3 1
gemboree-0.1.2.gem 3 1
gemmer-0.1.2.gem 3 1
geoinfo-0.0.5.gem 3 1
geti-1.5.0.gem 3 1
gherkin-2.11.6.gem 3 1
ghp-0.0.2.gem 3 1
ghp-0.0.5.gem 3 1
git_versioner-0.1.1.gem 3 1
gitpusher-0.0.4.gem 3 1
gollum_rails-0.0.2.2.gem 3 1
gollum_rails-0.0.2.5.gem 3 1
gollum_rails-0.0.2.7.gem 3 1
gollum_rails-0.0.2.gem 3 1
graph_mediator-0.2.3.gem 3 1
graymatter-0.1.0.gem 3 1
green-0.1.1.gem 3 1
gstreamer-1.2.1.gem 3 1
gtk2-1.2.1.gem 3 1
gtk3-1.2.1.gem 3 1
gtksourceview2-1.2.1.gem 3 1
gtksourceview3-1.2.1.gem 3 1
guard-1.6.2.gem 3 1
h2ocube_rails_assets-0.0.9.gem 3 1
hactor-0.0.2.gem 3 1
haml_coffee_assets-1.11.1.gem 3 1
hello_sign-0.0.5.gem 3 1
hello_sign-0.2.0.gem 3 1
help_spot-0.2.1.gem 3 1
home-0.215.43.gem 3 1
homeostasis-0.0.15.gem 3 1
httparty-0.10.1.gem 3 1
i18n_country_select-1.0.15.gem 3 1
ido-0.0.1.gem 3 1
imposition-0.9.3.gem 3 1
imposition-0.9.4.2.gem 3 1
imposition-0.9.4.gem 3 1
inflation-0.1.0.gem 3 1
inflation-0.1.2.gem 3 1
infopark_cloud_connector-6.8.2.7.128454143.gem 3 1
inkwell-0.0.1.gem 3 1
instagram_geo-0.8.7.gem 3 1
intercom-0.0.11.gem 3 1
iso639-1.1.0.gem 3 1
iso_country_codes-0.4.0.gem 3 1
jbundler-0.4.1.gem 3 1
jenkins_api_client-0.7.1.gem 3 1
jently-1.0.4.gem 3 1
jetstream-3.1.16.gem 3 1
jetstream-3.1.18.gem 3 1
jetstream-3.1.2.gem 3 1
jetstream-3.1.21.gem 3 1
jetstream-3.1.23.gem 3 1
jetstream-3.1.25.gem 3 1
jetstream-3.1.3.gem 3 1
jetstream-3.1.5.gem 3 1
jplugin-0.0.8.gem 3 1
kevorkian-1.0.0.gem 3 1
keyman-1.0.0.gem 3 1
killbill-1.0.0.gem 3 1
konacha-2.4.0.gem 3 1
kvom-6.8.2.7.128454143.gem 3 1
lanyrd-0.1.1.gem 3 1
lazier-2.0.0.gem 3 1
lazier-2.1.1.gem 3 1
lazier-2.3.0.gem 3 1
lazier-2.4.0.gem 3 1
le-2.1.4.gem 3 1
lederhosen-2.0.5.gem 3 1
leofs_manager_client-0.2.15.gem 3 1
libmediainfo-1.0.1.gem 3 1
listable-0.1.2.gem 3 1
livetask-0.0.4.gem 3 1
localyzed-0.0.2.6.gem 3 1
localyzed-0.0.2.8.gem 3 1
localyzed-0.0.3.0.gem 3 1
localyzed-0.0.3.2.gem 3 1
localyzed-0.0.3.4.gem 3 1
lol_dba-1.4.2.gem 3 1
lwes-0.8.4.gem 3 1
mail_room-0.0.2.gem 3 1
mail_room-0.1.0.gem 3 1
maniok_bdd-0.0.0.gem 3 1
maths-0.0.14.gem 3 1
messQ-0.0.1.gem 3 1
metamagic-2.0.5.gem 3 1
metriksd-0.5.3.gem 3 1
midwire_common-0.1.1.gem 3 1
mikesstudiogame-1.0.2.gem 3 1
milkode-0.9.7.gem 3 1
minecraft_api-0.0.1.gem 3 1
missing_validators-0.1.1.gem 3 1
mongo_request_logger-0.2.0.gem 3 1
mongoid_integer_id-0.1.gem 3 1
mongoid_retry-0.0.2.gem 3 1
moob-0.3.8.gem 3 1
motionscan-0.0.1.gem 3 1
mrubymix-0.0.2.gem 3 1
multicity-1.1.0.gem 3 1
multiparameter_assignable_attr-0.2.2.gem 3 1
multiparameter_assignable_attr-0.2.gem 3 1
musicbrainz-0.7.4.gem 3 1
mustache_render-0.0.15.gem 3 1
mustache_render-0.0.21.gem 3 1
mysql_truck-0.6.1.gem 3 1
nagios_check-0.3.0.gem 3 1
namebox-0.2.1.gem 3 1
narray-0.6.0.5.gem 3 1
negroku-0.0.2.gem 3 1
neo4j_server-1.8.1.gem 3 1
net_http_timeout_errors-0.1.0.gem 3 1
netconf-0.2.5.gem 3 1
nightfury-0.4.3.gem 3 1
nike_v2-0.1.0.gem 3 1
no_notifier_needed-2.0.0.gem 3 1
no_notifier_needed-2.0.2.gem 3 1
no_notifier_needed-2.0.4.gem 3 1
nobel-0.0.2.gem 3 1
normal-0.0.0.gem 3 1
notroff-0.3.5.gem 3 1
nyanko-0.0.1.gem 3 1
nyanko-0.0.4.gem 3 1
nyanko-0.0.6.gem 3 1
oddb2xml-1.2.5.gem 3 1
origami-1.2.5.gem 3 1
osmn-0.1.2.gem 3 1
osxsub-0.1.1.gem 3 1
otto-0.3.2.gem 3 1
page_navigation-0.2.gem 3 1
page_navigation-0.4.gem 3 1
paloma-1.2.4.gem 3 1
parallel_work-0.0.1.gem 3 1
parallelized_specs-0.3.95.gem 3 1
parallelized_specs-0.3.97.gem 3 1
party_foul-0.4.0.gem 3 1
pathstring-0.0.3.gem 3 1
payr-1.0.5.gem 3 1
pdftk-0.0.1.gem 3 1
pdftk-0.0.3.gem 3 1
perpetuity-0.4.4.gem 3 1
phonie-1.0.2.gem 3 1
pi_piper-1.3.gem 3 1
pirate_metrics_agent-0.1.3.gem 3 1
plek-1.1.0.gem 3 1
pmap-1.0.0.gem 3 1
poniard-0.0.2.gem 3 1
power_enum-1.0.1.gem 3 1
primo-0.0.2.gem 3 1
primo-0.0.4.gem 3 1
primo-0.0.6.gem 3 1
protobuf-2.6.4.gem 3 1
qaol-1.1.1.gem 3 1
qpid_messaging-0.18.3.gem 3 1
questionable_surveys-0.1.0.gem 3 1
quick_config-0.0.1.gem 3 1
qwester-0.0.5.gem 3 1
rabbit_jobs-0.3.gem 3 1
rails_account_location-1.0.1.gem 3 1
rails_admin_phone_number_field-0.0.1.gem 3 1
rails_finder-0.0.1.gem 3 1
rails_kindeditor_qiniu-0.0.1.gem 3 1
rails_kindeditor_qiniu-0.0.3.gem 3 1
ratchetio-0.6.0.gem 3 1
ratistics-0.1.0.gem 3 1
rbatch-1.7.0.gem 3 1
rcp-0.0.0.gem 3 1
rcp-0.0.2.gem 3 1
rcp-1.0.0.gem 3 1
recomiendo-0.1.0.gem 3 1
recordselect_vho-3.0.204.gem 3 1
recordselect_vho-3.0.206.gem 3 1
require_reloader-0.1.5.gem 3 1
ress-0.0.2.gem 3 1
ress-0.0.4.gem 3 1
ress-0.0.6.gem 3 1
restrack-1.8.2.gem 3 1
rethtool-0.0.3.gem 3 1
reverse_markdown-0.4.3.gem 3 1
reviewed_braai-0.1.0.gem 3 1
rgrmux-0.0.1.gem 3 1
right_support-2.6.16.gem 3 1
rjoystick-0.1.1.gem 3 1
roots-1.0.0.gem 3 1
rroonga-2.1.3.gem 3 1
rspectacular-0.3.0.gem 3 1
rspectacular-0.5.0.gem 3 1
rusen-0.0.2.gem 3 1
russian_phone-0.1.1.gem 3 1
s3cp-1.1.13.gem 3 1
s3cp-1.1.15.gem 3 1
safe_yaml-0.5.1.gem 3 1
scad4r-0.1.0.gem 3 1
scene-0.0.0.gemkeep.gem 2
scrapzirra-0.0.2.gem 3 1
scrapzirra-0.0.4.gem 3 1
scron-1.0.2.gem 3 1
seeing_is_believing-0.0.4.gem 3 1
sensr-1.0.0.gem 3 1
sensr-1.0.2.gem 3 1
sequel_bitemporal-0.6.4.gem 3 1
shared_mustache-0.0.1.gem 3 1
simple_attribute_mapper-0.0.3.gem 3 1
simple_changelog-0.0.12.gem 3 1
sindex-0.1.6.gem 3 1
sketchily-0.0.0.gem 3 1
skylight-0.0.2.gem 3 1
slicer-0.0.2.gem 3 1
slimmer-3.10.1.gem 3 1
smartguard-0.3.11.gem 3 1
smartguard-0.3.8.gem 3 1
snail-0.7.1.gem 3 1
social_stream-1.1.2.gem 3 1
sometimes-0.0.1.gem 3 1
spanish_number-0.1.0.gem 3 1
split-0.5.0.gem 3 1
spreewald-0.4.0.gem 3 1
spring-0.0.5.gem 3 1
spud_core-0.9.14.gem 3 1
sshkit-0.0.19.gem 3 1
stasis-0.2.0.gem 3 1
stasis_scaffolding-0.0.1.gem 3 1
stasis_scaffolding-0.1.0.gem 3 1
stipe-0.0.5.7.2.gem 3 1
storey-0.3.5.gem 3 1
storey-0.4.0.gem 3 1
stratum-0.2.2.gem 3 1
streamio-1.0.8.gem 3 1
stretcher-1.1.3.gem 3 1
structures-0.0.1.gem 3 1
studio_fame_game-0.0.1.gem 3 1
sufia-0.0.8.gem 3 1
sumodev_deploy-0.3.gem 3 1
svg_charts-1.01.gem 3 1
symbiont-0.1.8.gem 3 1
table_print-1.1.0.gem 3 1
tailor-1.1.4.gem 3 1
tassadar-0.3.0.gem 3 1
tbk-0.9.2.gem 3 1
tddium-1.7.1.gem 3 1
tekeya-0.0.9.gem 3 1
thetvdb-0.1.gem 3 1
ticketable-0.1.0.gem 3 1
tinted_tags-0.0.2.gem 3 1
tjstyle-0.0.1.gem 3 1
tjstyle-0.9.1.gem 3 1
tkh_admin_panel-0.3.1.gem 3 1
tkh_admin_panel-0.3.gem 3 1
tkh_authentication-0.1.3.gem 3 1
tkh_content-0.3.1.gem 3 1
tkh_illustrations-0.2.gem 3 1
tkh_toolbox-0.0.1.gem 3 1
to_hipchat-0.0.1.gem 3 1
to_source-0.2.18.gem 3 1
totem-0.0.1.gem 3 1
town-0.2.0.gem 3 1
town-0.2.2.gem 3 1
town-0.2.4.gem 3 1
town-0.2.6.gem 3 1
town-0.2.8.gem 3 1
town-0.3.0.gem 3 1
town-0.3.2.gem 3 1
town-0.3.4.gem 3 1
town-0.3.6.gem 3 1
town-0.3.8.gem 3 1
town-0.4.0.gem 3 1
town-0.4.2.gem 3 1
town-0.4.4.gem 3 1
tp-0.3.0.gem 3 1
tp-0.4.0.gem 3 1
tp-0.5.0.gem 3 1
trema-0.3.5.gem 3 1
tribe_em-0.0.2.gem 3 1
trumpet-0.0.0.gem 3 1
tumblr_client-0.6.9.gem 3 1
twog-0.3.2.gem 3 1
uploads-0.0.1.gem 3 1
veritrans-1.2.0.gem 3 1
vgh-0.2.1.gem 3 1
videojoiner-0.0.4.gem 3 1
viki_utils-0.0.3.gem 3 1
vineco-0.0.1.gem 3 1
vte3-1.2.1.gem 3 1
wacky-0.1.2.gem 3 1
wacky-0.1.4.gem 3 1
waffleiron-0.1.4.gem 3 1
webserver-0.0.1.gem 3 1
whitecms_news-0.0.2.gem 3 1
whitecms_news-0.0.4.gem 3 1
whitepaper-0.0.2.gem 3 1
who-0.0.1.gem 3 1
whoops_rails_logger-0.1.16.gem 3 1
wicked_pdf_standalone-0.0.3.gem 3 1
widgets-0.0.0.gem 3 1
widgets-0.1.0.gem 3 1
with_advisory_lock-0.0.3.gem 3 1
workless-1.1.2.gem 3 1
xrono-1.0.3.gem 3 1
yamwow-0.0.5.gem 3 1
yamwow-0.0.7.gem 3 1
yardstick-0.9.1.gem 3 1
yeti-0.3.5.gem 3 1
yeti-0.3.7.gem 3 1
zsh_dots-0.6.0.gem 3 1

[drbrain]

Good catch!

Can you check my work here:

https://github.com/rubygems/rubygems-verification/blob/master/redis_verify.rb#L87

All the results above except scene-0.0.0.gemkeep.gem (a prerelease) have three SHA checksums with only one entry in the checksum set, which makes me think that my test on line 87 is for four matching SHA checksums (S3, source A, source B, source C), not three (S3, source A, source B)

[heathd]

I'm afraid I'm not familiar with redis, so it's hard for me to verify the behaviour of the code. I'll try to write some pseudo code to explain my understanding:

class GemChecksumStore
  def initialize
    @gem_checksums = {}
  end

  def add(gemname, checksum)
    @gem_checksums[gemname] ||= []
    @gem_checksums[gemname] << checksum
  end

  def fetch(gemname)
    @gem_checksums[gemname] || []
  end

  def set_cardinality(gemname)
    fetch(gemname).uniq.size
  end

  def count(gemname)
    fetch(gemname).size
  end

  def set_size(gemname, desired_checksum)
    fetch(gemname).select {|checksum| checksum == desired_checksum}.count
  end
end

given the above class, I think that you are effectively doing:

class GemChecksumStore
  def verified?(gemname)
    case count(gemname)
    when 0, 1, 2
      false
    else
      count(gemname) - set_cardinality(gemname) > 2
    end
  end
end

wheras I think you need to actually check the size of the set for the particular checksum in the rubygems-sha512.S3.txt file, in other words:

class GemChecksumStore
  def verified?(gemname, s3_sha)
    set_size(gemname, s3_sha) > 2
  end
end

[drbrain]

Your above code matches the redis operations.

You're right, I'll update the redis code later today or tomorrow.

I was assuming that, since I separately verified the checksums against at least one mirror, I could assume multiple matches, but this is probably unsafe.

[drbrain]

I updated redis_verify per your suggestions and the list of unverified got smaller, but four prereleases were added as unverified.

Per your command the list still contains only prerelease gems.

See the commit message above for full details, I'll reopen if you have further questions.