tarsupervoxels: /tarfile endpoint returns incomplete results

[stuarteberg]

Today I loaded meshes for body 267551639 into the following tarsupervoxels instance: http://emdata3:8900/api/node/2053c1a64f254961874d91407e7301e3/segmentation_sv_meshes/info

I loaded 1969 supervoxels into the instance, via a single call to the /load endpoint. (The body has a total of 1975 supervoxels, but a few were too small to be detected in scale-1 segmentation.)

However, when I attempt to fetch the supervoxel meshes via the /tarfile endpoint, I get back only 359 meshes. I know the mesh files were uploaded and stored, because I can manually inspect the contents of the file store in /nvme0/filestore/hemi-blobs, and the "missing" supervoxel files are indeed present.

(I'll attach some example calls and logs in my next comment.)

[stuarteberg]

$ curl -s http://emdata3:8900/api/node/2053/segmentation_sv_meshes/tarfile/267551639 > 267551639.tar
$ mkdir 267551639
$ cd 267551639
$ tar -xf ../267551639.tar
$ ls | wc
      95      95    1344

EDIT: Something weird is going on... Now when I fetch the tarball, I get even fewer meshes. Only 95, as shown above. But I got more the last time I tried.

EDIT2: It seems that I get a different number of supervoxels each time I request the tarball. Strange...

Here's the list of (some of) the supervoxels that are MISSING from the tarball:

Click here for missing supervoxels

``` {421068800, 484847617, 1044062213, 1168871429, 981327879, 948600854, 297947158, 890912792, 1045098519, 583036953, 984047645, 798490655, 1077252128, 1168871459, 1168875560, 1106145321, 1106145324, 886186031, 421072943, 913481778, 1326899251, 1105813559, 922284091, 854835262, 1106145343, 482447426, 854831171, 798490696, 486219855, 921960527, 611041360, 1013719124, 581353564, 921960545, 798494817, 767455331, 1326899302, 1105813607, 423800935, 951644265, 645439597, 951967856, 854831222, 705044600, 1326899321, 548630648, 1106149503, 767463555, 676802696, 983015563, 614080653, 1015419021, 883474576, 1012695187, 1292656787, 1046462615, 767463583, 486219937, 551657634, 551657636, 614080679, 700010669, 614080686, 583033007, 890912944, 706080944, 546226361, 1292656825, 614084800, 614080705, 948609221, 829526215, 551657676, 551661772, 700010702, 551993556, 614080728, 890912991, 1105813730, 798494950, 421400807, 611266797, 1043402991, 981328112, 951972079, 1014845682, 1043402995, 984051968, 614080769, 1036943618, 767463685, 766898439, 581361928, 735863052, 614080780, 1107493134, 582705423, 390365455, 360689936, 1074774292, 614080797, 614080804, 918581540, 981324070, 580002090, 922288430, 1107493167, 581357871, 1012695346, 706081077, 551993657, 1575178554, 984047933, 922435902, 1044066623, 1106145601, 583033155, 642638152, 484852040, 831955274, 675045708, 859554126, 735863120, 676802896, 668324177, 1292648789, 1075106137, 484852059, 483823965, 360690015, 551993703, 542798183, 451744104, 1014845809, 890917233, 951972218, 1107497342, 982684031, 1074774401, 1230229893, 886198663, 1075114377, 674005387, 855159181, 706081167, 487907731, 360690070, 735863198, 481751456, 674009505, 577274275, 361648553, 769544617, 890913197, 551661999, 668324272, 859877807, 675045815, 361648570, 1075114431, 983015884, 577274317, 767463888, 551666130, 614080980, 674005462, 951976407, 545194456, 890921433, 982684120, 481751513, 551662046, 829534691, 1014063591, 1045098985, 636953067, 583033324, 891400683, 886198765, 551666167, 981328383, 981328384, 981324287, 699372032, 455176706, 1044070925, 455176727, 551670302, 920932899, 614081060, 551993893, 481751588, 551670311, 981328427, 1016549933, 982684205, 544162352, 829538865, 951644719, 551670321, 917234232, 551662138, 583033403, 860025403, 546230849, 551666242, 642413123, 583033413, 1137844807, 1077256776, 981324360, 951968330, 706081356, 614081107, 950288981, 455176789, 455176791, 829317724, 1045430878, 674005600, 981324392, 544162411, 486220397, 486220398, 822760046, 706081391, 551666289, 1045430898, 981328504, 829538936, 886198905, 984384124, 642413180, 704176766, 770593406, 706081414, 1014846087, 767459977, 950289035, 860025485, 890913425, 642638481, 517595795, 886198933, 799310488, 1106145945, 981324442, 645780125, 890917534, 984384159, 699372189, 920933025, 699372199, 642417321, 645780137, 951968427, 1046229674, 770593450, 889565869, 330289842, 517825205, 1014063809, 829539009, 1106145985, 889565893, 455414469, 860025543, 389333705, 389333706, 798839499, 674009805, 829539022, 604873425, 1016550097, 951968468, 950289117, 455176931, 1199936228, 1106146025, 886199018, 419332843, 766436075, 1261617906, 515187443, 330289908, 515187452, 637616892, 770593534, 614081292, 735548172, 891728656, 983823124, 767795991, 359662359, 922768151, 1137521432, 889565980, 891253534, 1014846243, 855171876, 1169212200, 1045435183, 548967221, 614409017, 829539129, 668980025, 886530876, 859558713, 889557818, 548967229, 855171908, 705389382, 736424787, 485184341, 891253589, 951976791, 855163740, 486552417, 886199141, 390366065, 951972723, 548971383, 583041912, 486552443, 576250752, 981345154, 982688644, 829539204, 886195078, 883815299, 485184389, 576250764, 645776269, 829318033, 389657492, 886199189, 1106486166, 1106146196, 1106486170, 1014064027, 855171996, 485184409, 859894681, 889557916, 984388508, 330285986, 1264829348, 917894053, 513835943, 359662504, 889557928, 984388523, 984388524, 1075442606, 389657515, 1106150325, 548967350, 981328823, 328283064, 389657529, 855163838, 328283072, 645465025, 580002755, 583041989, 981328837, 516228039, 889562056, 890921927, 981328841, 583041990, 389657539, 922301385, 1043739599, 889557970, 982688723, 1044075474, 642417621, 550646739, 951653335, 576250840, 981328859, 1169216475, 550646747, 1169212385, 330613730, 358286307, 978625508, 645784548, 361657313, 920933348, 917894117, 921412585, 389669865, 642413540, 580002799, 890930164, 798831610, 917894138, 917898236, 455513086, 922301439, 981664768, 889558018, 1106146309, 330290184, 886195210, 580002828, 1066959886, 890917906, 677143571, 389669911, 645784603, 419341341, 981992480, 951649312, 452785191, 1012700200, 548971571, 984388663, 1106482235, 797475904, 886199360, 1044067395, 1169212484, 645784645, 984392773, 581698637, 673789007, 389657679, 645788756, 1044067414, 1044079712, 546567264, 922301538, 1108968547, 581698660, 704513125, 580002915, 1044067428, 950293610, 798282861, 1137521775, 981664880, 581694587, 981668988, 984388733, 550646911, 580338824, 645780617, 677147788, 550646925, 516228241, 677151893, 548967575, 614421657, 486552730, 890926237, 951985310, 548967585, 581698722, 645465250, 920941739, 516228268, 917574829, 420709550, 981664944, 921412789, 797476024, 1044403390, 1106490561, 581698755, 891253958, 920941769, 886203594, 828839116, 920937677, 920933583, 645461200, 952308949, 920937686, 606250200, 580007139, 984388836, 920610022, 484164838, 645780710, 705385707, 860218604, 891262188, 735401197, 637949169, 580007155, 637949170, 645461240, 359671038, 1137853696, 767800574, 859878655, 1013028096, 358622470, 645780743, 948937992, 389657863, 948937997, 580007182, 951981327, 450721038, 1137857810, 920937747, 581698843, 389670172, 645465381, 359671079, 640029993, 484164909, 797476146, 917574962, 921412916, 891262261, 886199606, 542803256, 1075115321, 580007228, 614421821, 917894462, 1075447105, 484164930, 920937796, 484156742, 645784904, 484164937, 1045763401, 952309069, 922305870, 828839247, 767804752, 917566804, 952309079, 1075115355, 1044071772, 981665122, 736417123, 547939686, 673793385, 983020906, 546571627, 1077609842, 393016693, 390710646, 580007287, 859874680, 1044407681, 424478082, 359671173, 917894534, 983020935, 1077609869, 951649681, 919254418, 547939737, 1075115420, 424478109, 891741601, 645465507, 483124644, 641938850, 1106486695, 550659495, 767464873, 484165038, 736757170, 388629939, 981669298, 645465527, 734840249, 860218816, 519611843, 886531524, 583050697, 359675337, 583050699, 1013036491, 423118286, 984389068, 1106490827, 580007371, 450721236, 951649749, 953013716, 486893018, 424478175, 614421983, 1075443171, 886855139, 519284198, 614421989, 891254247, 985515497, 1075447275, 640030188, 611378674, 484156915, 1044403704, 424478201, 891258361, 605550080, 390706690, 917894664, 951752204, 1075447311, 580339216, 832300560, 645785106, 736757267, 359675410, 550655506, 1106490904, 1075447322, 1075447324, 770934299, 517928478, 891262501, 951752234, 450717227, 1014400557, 1077610034, 1075447352, 388630073, 1044403770, 891262524, 953017919, 1014404671, 637949504, 824137283, 952309316, 983823946, 547939916, 921744975, 642750034, 951981651, 1168893527, 921269851, 981665375, 949274209, 550659683, 1075447398, 824137322, 574174827, 642754157, 705390189, 547944048, 951981681, 577615475, 1013032565, 828843637, 824137333, 574174837, 797816443, 949274244, 483128965, 1013040774, 828843661, 1075447438, 605554320, 389674640, 1482073749, 1075451542, 606582424, 388638361, 645465755, 1013032606, 1013036707, 1482073771, 769889963, 951981742, 1075447471, 483129010, 1168893621, 953349816, 517928634, 951981754, 640030396, 1075451580, 921753280, 329664194, 860223175, 424478412, 517932749, 515536588, 1044412111, 1046234839, 1138185945, 1012704988, 1106487014, 1168893670, 1013036774, 1168893673, 770938602, 917907180, 797816558, 614430447, 1015768819, 1075779315, 359671543, 642754298, 550987515, 829335294, 1014163198, 951752443, 637634305, 1044403965, 983127812, 920950533, 389674759, 519608080, 613394194, 918230801, 886859541, 917907226, 606582556, 801978140, 1168893728, 357594914, 1044416291, 614745895, 519612202, 424478508, 605554477, 1138181941, 545212214, 824141622, 860219197, 766215997, 823105346, 455853892, 1015195462, 891266892, 359675724, 419682124, 1014404940, 483129168, 543139668, 889898837, 328623962, 359675738, 605550431, 1075447650, 424478563, 823105378, 1138181990, 886859623, 739575658, 577615728, 858859377, 1044080498, 550987635, 983029617, 327591797, 704513909, 550987638, 735733625, 455858044, 1075447681, 545212296, 391047051, 550664075, 483129230, 739575699, 891271060, 516904853, 798619543, 891262875, 798623645, 419682205, 886859680, 1044416416, 516904865, 388642725, 454490023, 483129257, 419682218, 798619566, 1013036977, 550987701, 613394358, 1138186167, 739575737, 1045764025, 891254714, 516904891, 515536830, 516904894, 891254720, 1013041091, 485529539, 798619591, 1014400968, 454490059, 517937099, 1138190285, 357599182, 1044416462, 767805393, 890238933, 549308374, 1229883351, 1075779545, 1044080604, 950962146, 1045768163, 517937123, 858859496, 1323337708, 642754542, 517937132, 858859504, 735737839, 886859764, 329664503, 984729593, 1138190330, 454494208, 606918660, 577615877, 767801350, 486893575, 606914567, 890238984, 673323020, 885516315, 978970651, 608995362, 606582817, 798623791, 891594802, 328628275, 419686456, 918566970, 1075451965, 419682368, 1137854544, 920950866, 388974678, 388974681, 1138190431, 918235234, 674130020, 735737960, 860711019, 613394540, 1137854575, 328640630, 485529726, 920950913, 1014397062, 550660233, 388630671, 574515350, 950630556, 1106487453, 797816990, 359671967, 983128225, 1229883555, 1014397092, 606914726, 1075456167, 516905126, 481061033, 735738028, 454494383, 1106491568, 357599410, 423454902, 978974906, 5813037244, 984729792, 1138190531, 1137862854, 482093255, 982005962, 978974931, 673323219, 920950997, 613394646, 358639831, 982005975, 1138194648, 579320029, 1075456222, 328640735, 738863327, 738859231, 1044412642, 891271398, 454494443, 454490348, 543144178, 921753847, 454494460, 1075456256, 486893825, 768145665, 1013041417, 738863369, 328964362, 1451038986, 923117844, 950962455, 1075452183, 886868249, 978970907, 328640797, 388966689, 357595427, 486893861, 1138190629, 921274661, 551311653, 550992170, 545216811, 738863411, 424818996, 860559679, 424819010, 674355528, 767801677, 768145743, 424819024, 643320145, 391055703, 424819032, 391055706, 606923104, 860219745, 424819039, 424819042, 770939236, 390011238, 984729959, 424819051, 481061246, 1044412802, 920951171, 860219782, 829188488, 735181194, 328964491, 735734159, 1014180240, 823437712, 424819089, 516905361, 770623893, 455858581, 391055769, 455858585, 390011291, 735181219, 424819108, 328640933, 419686822, 424819112, 949275050, 390007210, 761743789, 860559790, 1075784116, 483142069, 516905397, 424819130, 550996413, 515549630, 483142077, 391055809, 735181244, 391055811, 516905416, 885516752, 1075780051, 735181268, 761743832, 735181276, 704141790, 919595490, 329665004, 390011375, 674458096, 573487601, 952093170, 573487603, 947931636, 921754101, 860219895, 891271676, 860563965, 514177533, 674355712, 328641025, 918567431, 738867719, 642279945, 768145931, 735734285, 514177551, 885520912, 485866000, 767805973, 546912793, 298244636, 516905507, 514505253, 360008238, 606923309, 704367152, 582367790, 575883827, 360012339, 704367161, 699021885, 580684351, 551643715, 699021895, 642280008, 829188680, 452106824, 452106823, 735181386, 829676108, 328637005, 574519880, 920951370, 328637012, 574515799, 514505305, 860559966, 388967009, 735734370, 1013373539, 1013377637, 1107155557, 1075788393, 829672042, 704146027, 1013377647, 1013377648, 953694831, 483142259, 704141945, 582363771, 484502141, 1171724932, 738867847, 1044421258, 1044753037, 543480480, 574511777, 636930723, 575883943, 829188776, 516905642, 704367275, 388967087, 767806130, 582367924, 860707509, 550996663, 482101944, 573491897, 762075831, 1075788469, 642284220, 1420004030, 950962880, 643099326, 613735102, 738867903, 673659589, 643320518, 798624455, 550996680, 983366348, 513477326, 1138195151, 513481423, 982006484, 674355925, 1138518743, 829184728, 917912280, 582687454, 860568289, 885521123, 738867940, 483142377, 891271913, 674458347, 982346478, 636930798, 358980340, 582367990, 949943032, 360008441, 762075901, 579320574, 358980351, 735181569, 579320578, 674355978, 1138518795, 980982539, 952093454, 704142092, 1169234708, 574520086, 485870362, 735734555, 551643930, 422427419, 551648030, 330005279, 330005280, 388967204, 1138522912, 952326947, 360012578, 543480616, 638290731, 391056174, 360012589, 983706416, 550996784, 452107058, 452434740, 574520117, 360012598, 950963001, 980982587, 452430653, 358980417, 642284354, 360012614, 984730439, 330005319, 330005321, 952326985, 582687559, 982006602, 1045879635, 574520151, 360012633, 452434783, 1138527073, 483142500, 891059048, 1138195305, 768146281, 1105472363, 392420203, 1044745071, 885517169, 574520177, 392330098, 885517173, 582368119, 828861305, 828861307, 739904383, 582691720, 762076041, 766782349, 360012689, 953695122, 515877779, 980986777, 890719144, 519949227, 513481645, 579320753, 1198848950, 1013377975, 762080183, 452107190, 859532219, 982338493, 580680638, 762080195, 516910020, 328981445, 582687689, 330005450, 513481681, 328981459, 579320789, 860236760, 579320792, 980986842, 829184985, 947936221, 358976480, 667986913, 392420325, 667986918, 859532265, 1198849003, 980986864, 889916407, 1138531320, 543480828, 980986880, 392420354, 672771075, 580680711, 859532298, 860572682, 983469070, 582695949, 762080275, 545557524, 1075788822, 707832855, 452434970, 982006817, 582695971, 704146473, 1107156008, 483142708, 861617205, 707832891, 983469115, 1292303422, 891059262, 980986951, 327937100, 392420428, 485870676, 360012884, 883133532, 1013386336, 514845793, 891059297, 360012899, 1105472612, 1013378151, 829529191, 1044757611, 952327279, 1012358257, 1199901811, 890575989, 1012358263, 890571906, 613739652, 1044745349, 950971525, 1013382280, 860568713, 1107160202, 950971534, 767589521, 1198845077, 551652501, 360012949, 1044757653, 513485977, 922094746, 1138523287, 360348824, 551652510, 980987039, 582696087, 829525161, 887532716, 982338733, 360021165, 953695406, 707833009, 828161197, 360012975, 645426358, 707833019, 1198849211, 980982971, 828161218, 828161221, 421072071, 1198849223, 360348872, 673672395, 798158028, 485870797, 798158027, 1012362443, 737111249, 1107160278, 421072087, 737111255, 885857501, 546925791, 707833059, 890572003, 707837156, 860568805, 1013386471, 328969447, 891059433, 486448362, 828161262, 707833071, 860568817, 673672436, 1013714165, 1138523380, 421072121, 485870856, 949611785, 358976779, 798489869, 1044757774, 358980881, 551656723, 860568863, 544849186, 642297127, 358980908, 1074437422, 610700594, 1036602675, 358976822, 423705160, 360348987, 982347071, 551648583, 611261770, 1015078221, 392420686, 513486159, 515886420, 891387222, 328981848, 1074437463, 829525340, 829197664, 1013386592, 798485861, 673107304, 983145834, 1013386603, 829197681, 705043837, 829525378, 543485314, 389320073, 980983179, 1044762001, 298581399, 573496731, 573492638, 1045876129, 330341793, 1261268388, 1036602791, 731045290, 1036602796, 330341806, 423800239, 611261871, 515878323, 918908340, 1036602807, 1015078329, 392420795, 1013714364, 484842943, 951639490, 798158277, 1012362693, 951639497, 951639499, 645115341, 1036602830, 1036602833, 1036602834, 1138523606, 551652823, 1261272540, 1044762080, 551652833, 673672677, 327941604, 551652840, 731045353, 860012011, 1044762092, 515878380, 731045358, 520613359, 767450609, 731045365, 330341881, 731045373, 357940738, 610700804, 360349188, 982351365, 645115399, 1074773514, 885861900, 645115405, 887201294, 327941652, 951631381, 542457367, 891387415, 887533083, 645115420, 514850331, 330341919, 829533731, 767118885, 513490469, 887201327, 673107504, 1107488309, 583028278, 1043066423, 486211128, 574856761, 574856762, 917229119, 613740096, 1013714498, 643419715, 329952838, 327941702, 642625098, 1043402324, 1140690517, 885857876, 612064858, 822083163, 918908510, 859536993, 890580580, 673111652, 357940836, 917229161, 1013714538, 887201387, 547262061, 1107496558, 583028334, 551329394, 547262069, 982343285, 543821433, 1261272699, 767450748, 420363905, 951303812, 330337925, 1074773644, 420363918, 1105809039, 984043152, 357944977, 705044110, 767450767, 642625172, 982683284, 551653014, 421072534, 391392920, 1107488409, 767119001, 330346139, 298585757, 1074437795, 676802214, 574856869, 546225830, 513490601, 982683309, 673107633, 1107492530, 357940917, 1074773688, 330337982, 357945023, 327945918, 551329471, 983707330, 699031231, 948264642, 980987590, 1168871111, 484843208, 1013714634, 944520907, 484847315, 731041494, 421072599, 1075101400, 357945050, 1075101403, 517254879, 330338023, 1107488490, 800579307, 360021744, 514846453, 548290298, 767450875, 484847356, 859533053, 1105805053, 579665669, 330338057, 860569353, 1014845196, 484851467, 484851471, 917237520, 357945108, 1014845206, 513490715, 551657251, 486453033, 360021802, 860569387, 829529901, 854830910, 767455039, 1045090111, 739209025, 484847426, 668323649, 548290366, 1013718850, 421072714, 484851530, 739209038, 739209045, 798486358, 674799447, 829529945, 920600414, 890580833, 548290401, 1107492709, 1075101546, 1075101549, 829534063, 700010352, 548290417, 829534066, 548851571, 767926130, 421072757, 551657338, 676798331, 421068672, 798502786, 1292308356, 1168875398, 1077251976, 551657357, 547262356, 859533205, 859541403, 982347680, 737456039, 542457767, 1045090221, 676802478, 984047536, 1199595440, 854826928, 548290482, 581345202, 482451391, 823783362, 854831043, 920264648, 854826954, 581345231, 548851664, 1292652498, 542453714, 484847578, 854831067, 1043726298, 676806621, 918908895, 548851681, 951644129, 982679522, 982679523, 542453734, 920592356, 1077252079, 579665903, 1077252082, 981327855, 421072884, 1106141169, 579661815, 1044062196, 614080511} ```

Spot-checking the first supervoxel in that list, it appears that it does indeed exist in the filestore:

$ ssh emdata3
$ find /nvme0/filestore/hemi-blobs -name "*421068800*"
/nvme0/filestore/hemi-blobs/66/16/91cd/v1-i34-421068800.drc

[DocSavage]

The /tarfile code should have returned with an error because having missing supervoxel files was considered an error. In any case, it would have resulted in premature termination of the tarfile. Will investigate why it didn't return with an error instead of the 200.

I have modified the code to be tolerant of missing meshes since that seems to be an unwritten part of the spec. The new code returns the proper # of files. So issue resolved with commit 6f9e2653.

[stuarteberg]

Thanks! Sorry for the incomplete spec...

[DocSavage]

I found the bug -- the error returned by the tar processing routine wasn't passed on but another err variable was, leading to the correct premature stoppage but not the propagation of the error. The current code fix doesn't have this bug so we should be good.