Some depth data missing?

[pwais]

Thank you for making the dataset available! I believe I downloaded the data correctly, but it appears that one sequence is missing depth data-- Train sequence 000 is missing 1138 depth frames. I'm pretty sure my download went fine, but I give details below. Should we just throw out sequence 000 or is the full depth data available?

Here's what 7z says about the archive:

Scanning the drive for archives:
1 file, 108827217362 bytes (102 GiB)

Listing archive: deepdeform_v1.7z

--
Path = deepdeform_v1.7z
Type = 7z
Physical Size = 108827217362
Headers Size = 5450645
Method = LZMA2:24
Solid = +
Blocks = 86

Get the file list:

7z l deepdeform_v1.7z > deepdeform_v1.7z.filelist.txt

Here's how I poked around and found the discrepancy... all the other sequences look good 👍

lines = open('deepdeform_v1.7z.filelist.txt').readlines() 
import re
from collections import defaultdict
for line in lines: 
  m = re.search('(train|test)\/seq(\d+)\/(color|depth)\/.*', line) 
  if m: 
    counts['-'.join(m.group(i) for i in (1, 2, 3))] += 1 
import pprint
pprint.pprint(counts)                                                                                                                    
>>> defaultdict(<class 'int'>,
            {'test-000-color': 421,
             'test-000-depth': 421,
             'test-001-color': 979,
             'test-001-depth': 979,
             'test-002-color': 629,
             'test-002-depth': 629,
             'test-003-color': 432,
             'test-003-depth': 432,
             'test-004-color': 549,
             'test-004-depth': 549,
             'test-005-color': 790,
             'test-005-depth': 790,
             'test-006-color': 856,
             'test-006-depth': 856,
             'test-007-color': 763,
             'test-007-depth': 763,
             'test-008-color': 894,
             'test-008-depth': 894,
             'test-009-color': 1023,
             'test-009-depth': 1023,
             'test-010-color': 1179,
             'test-010-depth': 1179,
             'test-011-color': 1064,
             'test-011-depth': 1064,
             'test-012-color': 962,
             'test-012-depth': 962,
             'test-013-color': 1026,
             'test-013-depth': 1026,
             'test-014-color': 869,
             'test-014-depth': 869,
             'test-015-color': 1029,
             'test-015-depth': 1029,
             'test-016-color': 1036,
             'test-016-depth': 1036,
             'test-017-color': 886,
             'test-017-depth': 886,
             'test-018-color': 902,
             'test-018-depth': 902,
             'test-019-color': 952,
             'test-019-depth': 952,
             'test-020-color': 781,
             'test-020-depth': 781,
             'test-021-color': 1017,
             'test-021-depth': 1017,
             'test-022-color': 996,
             'test-022-depth': 996,
             'test-023-color': 1185,
             'test-023-depth': 1185,
             'test-024-color': 1130,
             'test-024-depth': 1130,
             'test-025-color': 1193,
             'test-025-depth': 1193,
             'test-026-color': 1254,
             'test-026-depth': 1254,
             'test-027-color': 967,
             'test-027-depth': 967,
             'test-028-color': 1049,
             'test-028-depth': 1049,
             'test-029-color': 1060,
             'test-029-depth': 1060,
             'train-000-color': 2632,
             'train-000-depth': 1494,
             'train-001-color': 4377,
             'train-001-depth': 4377,
             'train-002-color': 2241,
             'train-002-depth': 2241,
             'train-003-color': 5688,
             'train-003-depth': 5688,
             'train-004-color': 1610,
             'train-004-depth': 1610,
             'train-005-color': 1203,
             'train-005-depth': 1203,
             'train-006-color': 6960,
             'train-006-depth': 6960,
             'train-007-color': 1660,
             'train-007-depth': 1660,
             'train-008-color': 9685,
             'train-008-depth': 9685,
             'train-009-color': 15797,
             'train-009-depth': 15797,
             'train-010-color': 1370,
             'train-010-depth': 1370,
             'train-011-color': 7791,
             'train-011-depth': 7791,
             'train-012-color': 6421,
             'train-012-depth': 6421,
             'train-013-color': 6867,
             'train-013-depth': 6867,
             'train-014-color': 6949,
             'train-014-depth': 6949,
             'train-015-color': 674,
             'train-015-depth': 674,
             'train-016-color': 801,
             'train-016-depth': 801,
             'train-017-color': 719,
             'train-017-depth': 719,
             'train-018-color': 843,
             'train-018-depth': 843,
             'train-019-color': 1089,
             'train-019-depth': 1089,
             'train-020-color': 885,
             'train-020-depth': 885,
             'train-021-color': 714,
             'train-021-depth': 714,
             'train-022-color': 787,
             'train-022-depth': 787,
             'train-023-color': 837,
             'train-023-depth': 837,
             'train-024-color': 788,
             'train-024-depth': 788,
             'train-025-color': 733,
             'train-025-depth': 733,
             'train-026-color': 551,
             'train-026-depth': 551,
             'train-027-color': 1272,
             'train-027-depth': 1272,
             'train-028-color': 380,
             'train-028-depth': 380,
             'train-029-color': 342,
             'train-029-depth': 342,
             'train-030-color': 255,
             'train-030-depth': 255,
             'train-031-color': 253,
             'train-031-depth': 253,
             'train-032-color': 259,
             'train-032-depth': 259,
             'train-033-color': 236,
             'train-033-depth': 236,
             'train-034-color': 563,
             'train-034-depth': 563,
             'train-035-color': 602,
             'train-035-depth': 602,
             'train-036-color': 453,
             'train-036-depth': 453,
             'train-037-color': 333,
             'train-037-depth': 333,
             'train-038-color': 492,
             'train-038-depth': 492,
             'train-039-color': 852,
             'train-039-depth': 852,
             'train-040-color': 613,
             'train-040-depth': 613,
             'train-041-color': 856,
             'train-041-depth': 856,
             'train-042-color': 709,
             'train-042-depth': 709,
             'train-043-color': 521,
             'train-043-depth': 521,
             'train-044-color': 643,
             'train-044-depth': 643,
             'train-045-color': 678,
             'train-045-depth': 678,
             'train-046-color': 520,
             'train-046-depth': 520,
             'train-047-color': 1051,
             'train-047-depth': 1051,
             'train-048-color': 682,
             'train-048-depth': 682,
             'train-049-color': 297,
             'train-049-depth': 297,
             'train-050-color': 331,
             'train-050-depth': 331,
             'train-051-color': 260,
             'train-051-depth': 260,
             'train-052-color': 226,
             'train-052-depth': 226,
             'train-053-color': 277,
             'train-053-depth': 277,
             'train-054-color': 249,
             'train-054-depth': 249,
             'train-055-color': 294,
             'train-055-depth': 294,
             'train-056-color': 225,
             'train-056-depth': 225,
             'train-057-color': 857,
             'train-057-depth': 857,
             'train-058-color': 935,
             'train-058-depth': 935,
             'train-059-color': 760,
             'train-059-depth': 760,
             'train-060-color': 833,
             'train-060-depth': 833,
             'train-061-color': 827,
             'train-061-depth': 827,
             'train-062-color': 540,
             'train-062-depth': 540,
             'train-063-color': 603,
             'train-063-depth': 603,
             'train-064-color': 803,
             'train-064-depth': 803,
             'train-065-color': 785,
             'train-065-depth': 785,
             'train-066-color': 908,
             'train-066-depth': 908,
             'train-067-color': 771,
             'train-067-depth': 771,
             'train-068-color': 674,
             'train-068-depth': 674,
             'train-069-color': 659,
             'train-069-depth': 659,
             'train-070-color': 783,
             'train-070-depth': 783,
             'train-071-color': 761,
             'train-071-depth': 761,
             'train-072-color': 750,
             'train-072-depth': 750,
             'train-073-color': 1018,
             'train-073-depth': 1018,
             'train-074-color': 768,
             'train-074-depth': 768,
             'train-075-color': 469,
             'train-075-depth': 469,
             'train-076-color': 553,
             'train-076-depth': 553,
             'train-077-color': 813,
             'train-077-depth': 813,
             'train-078-color': 779,
             'train-078-depth': 779,
             'train-079-color': 841,
             'train-079-depth': 841,
             'train-080-color': 987,
             'train-080-depth': 987,
             'train-081-color': 684,
             'train-081-depth': 684,
             'train-082-color': 909,
             'train-082-depth': 909,
             'train-083-color': 1149,
             'train-083-depth': 1149,
             'train-084-color': 1150,
             'train-084-depth': 1150,
             'train-085-color': 1096,
             'train-085-depth': 1096,
             'train-086-color': 578,
             'train-086-depth': 578,
             'train-087-color': 527,
             'train-087-depth': 527,
             'train-088-color': 249,
             'train-088-depth': 249,
             'train-089-color': 778,
             'train-089-depth': 778,
             'train-090-color': 336,
             'train-090-depth': 336,
             'train-091-color': 256,
             'train-091-depth': 256,
             'train-092-color': 540,
             'train-092-depth': 540,
             'train-093-color': 400,
             'train-093-depth': 400,
             'train-094-color': 916,
             'train-094-depth': 916,
             'train-095-color': 914,
             'train-095-depth': 914,
             'train-096-color': 878,
             'train-096-depth': 878,
             'train-097-color': 755,
             'train-097-depth': 755,
             'train-098-color': 966,
             'train-098-depth': 966,
             'train-099-color': 800,
             'train-099-depth': 800,
             'train-100-color': 831,
             'train-100-depth': 831,
             'train-101-color': 184,
             'train-101-depth': 184,
             'train-102-color': 762,
             'train-102-depth': 762,
             'train-103-color': 287,
             'train-103-depth': 287,
             'train-104-color': 232,
             'train-104-depth': 232,
             'train-105-color': 334,
             'train-105-depth': 334,
             'train-106-color': 501,
             'train-106-depth': 501,
             'train-107-color': 778,
             'train-107-depth': 778,
             'train-108-color': 919,
             'train-108-depth': 919,
             'train-109-color': 723,
             'train-109-depth': 723,
             'train-110-color': 298,
             'train-110-depth': 298,
             'train-111-color': 838,
             'train-111-depth': 838,
             'train-112-color': 164,
             'train-112-depth': 164,
             'train-113-color': 181,
             'train-113-depth': 181,
             'train-114-color': 169,
             'train-114-depth': 169,
             'train-115-color': 417,
             'train-115-depth': 417,
             'train-116-color': 735,
             'train-116-depth': 735,
             'train-117-color': 296,
             'train-117-depth': 296,
             'train-118-color': 378,
             'train-118-depth': 378,
             'train-119-color': 376,
             'train-119-depth': 376,
             'train-120-color': 282,
             'train-120-depth': 282,
             'train-121-color': 299,
             'train-121-depth': 299,
             'train-122-color': 239,
             'train-122-depth': 239,
             'train-123-color': 180,
             'train-123-depth': 180,
             'train-124-color': 479,
             'train-124-depth': 479,
             'train-125-color': 605,
             'train-125-depth': 605,
             'train-126-color': 860,
             'train-126-depth': 860,
             'train-127-color': 686,
             'train-127-depth': 686,
             'train-128-color': 789,
             'train-128-depth': 789,
             'train-129-color': 768,
             'train-129-depth': 768,
             'train-130-color': 251,
             'train-130-depth': 251,
             'train-131-color': 321,
             'train-131-depth': 321,
             'train-132-color': 679,
             'train-132-depth': 679,
             'train-133-color': 618,
             'train-133-depth': 618,
             'train-134-color': 669,
             'train-134-depth': 669,
             'train-135-color': 581,
             'train-135-depth': 581,
             'train-136-color': 574,
             'train-136-depth': 574,
             'train-137-color': 581,
             'train-137-depth': 581,
             'train-138-color': 843,
             'train-138-depth': 843,
             'train-139-color': 779,
             'train-139-depth': 779,
             'train-140-color': 625,
             'train-140-depth': 625,
             'train-141-color': 681,
             'train-141-depth': 681,
             'train-142-color': 785,
             'train-142-depth': 785,
             'train-143-color': 834,
             'train-143-depth': 834,
             'train-144-color': 942,
             'train-144-depth': 942,
             'train-145-color': 574,
             'train-145-depth': 574,
             'train-146-color': 795,
             'train-146-depth': 795,
             'train-147-color': 661,
             'train-147-depth': 661,
             'train-148-color': 825,
             'train-148-depth': 825,
             'train-149-color': 1022,
             'train-149-depth': 1022,
             'train-150-color': 612,
             'train-150-depth': 612,
             'train-151-color': 509,
             'train-151-depth': 509,
             'train-152-color': 1154,
             'train-152-depth': 1154,
             'train-153-color': 854,
             'train-153-depth': 854,
             'train-154-color': 731,
             'train-154-depth': 731,
             'train-155-color': 1035,
             'train-155-depth': 1035,
             'train-156-color': 1029,
             'train-156-depth': 1029,
             'train-157-color': 1060,
             'train-157-depth': 1060,
             'train-158-color': 1369,
             'train-158-depth': 1369,
             'train-159-color': 1389,
             'train-159-depth': 1389,
             'train-160-color': 1137,
             'train-160-depth': 1137,
             'train-161-color': 866,
             'train-161-depth': 866,
             'train-162-color': 897,
             'train-162-depth': 897,
             'train-163-color': 1134,
             'train-163-depth': 1134,
             'train-164-color': 1016,
             'train-164-depth': 1016,
             'train-165-color': 1176,
             'train-165-depth': 1176,
             'train-166-color': 793,
             'train-166-depth': 793,
             'train-167-color': 1319,
             'train-167-depth': 1319,
             'train-168-color': 885,
             'train-168-depth': 885,
             'train-169-color': 1098,
             'train-169-depth': 1098,
             'train-170-color': 1207,
             'train-170-depth': 1207,
             'train-171-color': 1103,
             'train-171-depth': 1103,
             'train-172-color': 1493,
             'train-172-depth': 1493,
             'train-173-color': 1019,
             'train-173-depth': 1019,
             'train-174-color': 989,
             'train-174-depth': 989,
             'train-175-color': 999,
             'train-175-depth': 999,
             'train-176-color': 840,
             'train-176-depth': 840,
             'train-177-color': 1007,
             'train-177-depth': 1007,
             'train-178-color': 1006,
             'train-178-depth': 1006,
             'train-179-color': 893,
             'train-179-depth': 893,
             'train-180-color': 946,
             'train-180-depth': 946,
             'train-181-color': 936,
             'train-181-depth': 936,
             'train-182-color': 955,
             'train-182-depth': 955,
             'train-183-color': 1067,
             'train-183-depth': 1067,
             'train-184-color': 1954,
             'train-184-depth': 1954,
             'train-185-color': 1315,
             'train-185-depth': 1315,
             'train-186-color': 1347,
             'train-186-depth': 1347,
             'train-187-color': 1229,
             'train-187-depth': 1229,
             'train-188-color': 1018,
             'train-188-depth': 1018,
             'train-189-color': 1044,
             'train-189-depth': 1044,
             'train-190-color': 1110,
             'train-190-depth': 1110,
             'train-191-color': 1335,
             'train-191-depth': 1335,
             'train-192-color': 1222,
             'train-192-depth': 1222,
             'train-193-color': 1252,
             'train-193-depth': 1252,
             'train-194-color': 801,
             'train-194-depth': 801,
             'train-195-color': 1427,
             'train-195-depth': 1427,
             'train-196-color': 1272,
             'train-196-depth': 1272,
             'train-197-color': 1149,
             'train-197-depth': 1149,
             'train-198-color': 1369,
             'train-198-depth': 1369,
             'train-199-color': 1459,
             'train-199-depth': 1459,
             'train-200-color': 1116,
             'train-200-depth': 1116,
             'train-201-color': 1047,
             'train-201-depth': 1047,
             'train-202-color': 1434,
             'train-202-depth': 1434,
             'train-203-color': 1254,
             'train-203-depth': 1254,
             'train-204-color': 1194,
             'train-204-depth': 1194,
             'train-205-color': 1389,
             'train-205-depth': 1389,
             'train-206-color': 1379,
             'train-206-depth': 1379,
             'train-207-color': 1438,
             'train-207-depth': 1438,
             'train-208-color': 1775,
             'train-208-depth': 1775,
             'train-209-color': 1269,
             'train-209-depth': 1269,
             'train-210-color': 1267,
             'train-210-depth': 1267,
             'train-211-color': 1258,
             'train-211-depth': 1258,
             'train-212-color': 1133,
             'train-212-depth': 1133,
             'train-213-color': 1279,
             'train-213-depth': 1279,
             'train-214-color': 1149,
             'train-214-depth': 1149,
             'train-215-color': 1697,
             'train-215-depth': 1697,
             'train-216-color': 1404,
             'train-216-depth': 1404,
             'train-217-color': 991,
             'train-217-depth': 991,
             'train-218-color': 1200,
             'train-218-depth': 1200,
             'train-219-color': 1188,
             'train-219-depth': 1188,
             'train-220-color': 1515,
             'train-220-depth': 1515,
             'train-221-color': 1374,
             'train-221-depth': 1374,
             'train-222-color': 1198,
             'train-222-depth': 1198,
             'train-223-color': 1077,
             'train-223-depth': 1077,
             'train-224-color': 999,
             'train-224-depth': 999,
             'train-225-color': 1062,
             'train-225-depth': 1062,
             'train-226-color': 1052,
             'train-226-depth': 1052,
             'train-227-color': 1014,
             'train-227-depth': 1014,
             'train-228-color': 981,
             'train-228-depth': 981,
             'train-229-color': 920,
             'train-229-depth': 920,
             'train-230-color': 901,
             'train-230-depth': 901,
             'train-231-color': 980,
             'train-231-depth': 980,
             'train-232-color': 945,
             'train-232-depth': 945,
             'train-233-color': 878,
             'train-233-depth': 878,
             'train-234-color': 933,
             'train-234-depth': 933,
             'train-235-color': 965,
             'train-235-depth': 965,
             'train-236-color': 962,
             'train-236-depth': 962,
             'train-237-color': 877,
             'train-237-depth': 877,
             'train-238-color': 898,
             'train-238-depth': 898,
             'train-239-color': 844,
             'train-239-depth': 844,
             'train-240-color': 1099,
             'train-240-depth': 1099,
             'train-241-color': 1009,
             'train-241-depth': 1009,
             'train-242-color': 963,
             'train-242-depth': 963,
             'train-243-color': 967,
             'train-243-depth': 967,
             'train-244-color': 846,
             'train-244-depth': 846,
             'train-245-color': 838,
             'train-245-depth': 838,
             'train-246-color': 813,
             'train-246-depth': 813,
             'train-247-color': 832,
             'train-247-depth': 832,
             'train-248-color': 820,
             'train-248-depth': 820,
             'train-249-color': 1351,
             'train-249-depth': 1351,
             'train-250-color': 1155,
             'train-250-depth': 1155,
             'train-251-color': 1116,
             'train-251-depth': 1116,
             'train-252-color': 1074,
             'train-252-depth': 1074,
             'train-253-color': 1187,
             'train-253-depth': 1187,
             'train-254-color': 1049,
             'train-254-depth': 1049,
             'train-255-color': 1080,
             'train-255-depth': 1080,
             'train-256-color': 1142,
             'train-256-depth': 1142,
             'train-257-color': 1065,
             'train-257-depth': 1065,
             'train-258-color': 1017,
             'train-258-depth': 1017,
             'train-259-color': 1097,
             'train-259-depth': 1097,
             'train-260-color': 1098,
             'train-260-depth': 1098,
             'train-261-color': 1045,
             'train-261-depth': 1045,
             'train-262-color': 1016,
             'train-262-depth': 1016,
             'train-263-color': 653,
             'train-263-depth': 653,
             'train-264-color': 713,
             'train-264-depth': 713,
             'train-265-color': 631,
             'train-265-depth': 631,
             'train-266-color': 795,
             'train-266-depth': 795,
             'train-267-color': 809,
             'train-267-depth': 809,
             'train-268-color': 991,
             'train-268-depth': 991,
             'train-269-color': 1528,
             'train-269-depth': 1528,
             'train-270-color': 904,
             'train-270-depth': 904,
             'train-271-color': 1171,
             'train-271-depth': 1171,
             'train-272-color': 983,
             'train-272-depth': 983,
             'train-273-color': 771,
             'train-273-depth': 771,
             'train-274-color': 981,
             'train-274-depth': 981,
             'train-275-color': 1140,
             'train-275-depth': 1140,
             'train-276-color': 1113,
             'train-276-depth': 1113,
             'train-277-color': 935,
             'train-277-depth': 935,
             'train-278-color': 1073,
             'train-278-depth': 1073,
             'train-279-color': 917,
             'train-279-depth': 917,
             'train-280-color': 1353,
             'train-280-depth': 1353,
             'train-281-color': 1690,
             'train-281-depth': 1690,
             'train-282-color': 1292,
             'train-282-depth': 1292,
             'train-283-color': 1511,
             'train-283-depth': 1511,
             'train-284-color': 449,
             'train-284-depth': 449,
             'train-285-color': 259,
             'train-285-depth': 259,
             'train-286-color': 290,
             'train-286-depth': 290,
             'train-287-color': 226,
             'train-287-depth': 226,
             'train-288-color': 272,
             'train-288-depth': 272,
             'train-289-color': 290,
             'train-289-depth': 290,
             'train-290-color': 285,
             'train-290-depth': 285,
             'train-291-color': 259,
             'train-291-depth': 259,
             'train-292-color': 368,
             'train-292-depth': 368,
             'train-293-color': 368,
             'train-293-depth': 368,
             'train-294-color': 218,
             'train-294-depth': 218,
             'train-295-color': 197,
             'train-295-depth': 197,
             'train-296-color': 155,
             'train-296-depth': 155,
             'train-297-color': 265,
             'train-297-depth': 265,
             'train-298-color': 198,
             'train-298-depth': 198,
             'train-299-color': 414,
             'train-299-depth': 414,
             'train-300-color': 346,
             'train-300-depth': 346,
             'train-301-color': 488,
             'train-301-depth': 488,
             'train-302-color': 496,
             'train-302-depth': 496,
             'train-303-color': 869,
             'train-303-depth': 869,
             'train-304-color': 792,
             'train-304-depth': 792,
             'train-305-color': 793,
             'train-305-depth': 793,
             'train-306-color': 452,
             'train-306-depth': 452,
             'train-307-color': 565,
             'train-307-depth': 565,
             'train-308-color': 753,
             'train-308-depth': 753,
             'train-309-color': 670,
             'train-309-depth': 670,
             'train-310-color': 642,
             'train-310-depth': 642,
             'train-311-color': 789,
             'train-311-depth': 789,
             'train-312-color': 813,
             'train-312-depth': 813,
             'train-313-color': 653,
             'train-313-depth': 653,
             'train-314-color': 553,
             'train-314-depth': 553,
             'train-315-color': 677,
             'train-315-depth': 677,
             'train-316-color': 566,
             'train-316-depth': 566,
             'train-317-color': 378,
             'train-317-depth': 378,
             'train-318-color': 804,
             'train-318-depth': 804,
             'train-319-color': 667,
             'train-319-depth': 667,
             'train-320-color': 646,
             'train-320-depth': 646,
             'train-321-color': 856,
             'train-321-depth': 856,
             'train-322-color': 456,
             'train-322-depth': 456,
             'train-323-color': 508,
             'train-323-depth': 508,
             'train-324-color': 481,
             'train-324-depth': 481,
             'train-325-color': 585,
             'train-325-depth': 585,
             'train-326-color': 508,
             'train-326-depth': 508,
             'train-327-color': 613,
             'train-327-depth': 613,
             'train-328-color': 627,
             'train-328-depth': 627,
             'train-329-color': 870,
             'train-329-depth': 870,
             'train-330-color': 725,
             'train-330-depth': 725,
             'train-331-color': 734,
             'train-331-depth': 734,
             'train-332-color': 754,
             'train-332-depth': 754,
             'train-333-color': 634,
             'train-333-depth': 634,
             'train-334-color': 385,
             'train-334-depth': 385,
             'train-335-color': 473,
             'train-335-depth': 473,
             'train-336-color': 459,
             'train-336-depth': 459,
             'train-337-color': 470,
             'train-337-depth': 470,
             'train-338-color': 317,
             'train-338-depth': 317,
             'train-339-color': 445,
             'train-339-depth': 445})

[pwais]

FWIW I also computed the SHA-1 of what I downloaded:

$ sha1sum deepdeform_v1.7z
f3b4aec3aee2243797148d27c0274bc5a09ae6e3  deepdeform_v1.7z

[wenbin-lin]

Same for me

[AljazBozic]

Sorry for the late response, thank you for noticing the problem! It was indeed a mistake from our side, some depth maps were somehow lost during the data preprocessing. We are preparing the corrected dataset version, with all depth maps present, and will notify everyone that downloaded the dataset via email this weekend. I'm sorry again for the inconvenience!

[pwais]

@AljazBozic any chance the missing scans might be offered in a separate download for those who have already downloaded the rest of the massive dataset?