search

hpcaitech / Open-Sora

Open-Sora: Democratizing Efficient Video Production for All
https://hpcaitech.github.io/Open-Sora/
Apache License 2.0
20.76k stars 1.97k forks source link

use parallel_apply in sampler get same result in each epoch #416

Closed oceanogeology closed 1 month ago

oceanogeology commented 1 month ago

https://github.com/hpcaitech/Open-Sora/blob/ea41df3d6cc5f389b6824572854d97fa9f7779c3/opensora/datasets/sampler.py#L56

this "parallel_apply" get same sample order in each epoch, but use "apply" get random sampler order.

use parallel_apply: [1, 3, 10, 11, 14, 16, 17, 30, 31, 35, 37, 44, 49, 51, 54, 56, 59, 64, 66, 69, 71, 74, 75, 77, 78, 81, 88, 98, 99, 100, 104, 105, 109, 112, 120, 122, 127, 128, 130, 134, 141, 143, 146, 154, 159, 161, 173, 179, 180, 181, 187, 189, 190, 194, 195, 196, 199, 201, 207, 213, 215, 218, 222, 223, 224, 230, 236, 241, 246, 251, 254, 261, 266, 268, 269, 273, 280, 284, 287, 292, 304, 308, 317, 319, 322, 323, 327, 329, 330, 332, 342, 344, 345, 348, 351, 357, 385, 386, 394, 400]

[1, 3, 10, 11, 14, 16, 17, 30, 31, 35, 37, 44, 49, 51, 54, 56, 59, 64, 66, 69, 71, 74, 75, 77, 78, 81, 88, 98, 99, 100, 104, 105, 109, 112, 120, 122, 127, 128, 130, 134, 141, 143, 146, 154, 159, 161, 173, 179, 180, 181, 187, 189, 190, 194, 195, 196, 199, 201, 207, 213, 215, 218, 222, 223, 224, 230, 236, 241, 246, 251, 254, 261, 266, 268, 269, 273, 280, 284, 287, 292, 304, 308, 317, 319, 322, 323, 327, 329, 330, 332, 342, 344, 345, 348, 351, 357, 385, 386, 394, 400]

use apply: [2, 4, 5, 7, 11, 15, 19, 24, 26, 27, 30, 36, 44, 54, 56, 58, 67, 72, 73, 76, 86, 87, 99, 100, 105, 109, 111, 125, 132, 134, 139, 148, 150, 164, 166, 175, 177, 178, 180, 184, 186, 191, 198, 206, 207, 220, 223, 227, 229, 232, 234, 238, 240, 246, 251, 254, 255, 257, 260, 263, 265, 268, 279, 283, 286, 298, 304, 305, 307, 308, 309, 310, 312, 320, 323, 325, 327, 336, 339, 346, 348, 354, 357, 363, 369, 375, 381, 382, 385, 387, 388, 393, 399, 402, 409, 410, 412, 418, 425, 439]

[4, 7, 12, 16, 19, 21, 25, 26, 32, 34, 35, 37, 45, 51, 65, 68, 73, 75, 80, 90, 93, 95, 110, 112, 115, 117, 125, 135, 140, 147, 149, 152, 156, 157, 162, 163, 165, 174, 181, 182, 191, 192, 202, 207, 208, 209, 210, 211, 213, 225, 231, 238, 240, 243, 247, 251, 254, 256, 259, 265, 282, 288, 294, 306, 308, 311, 312, 321, 323, 324, 326, 332, 335, 341, 343, 347, 350, 353, 363, 368, 370, 371, 372, 374, 377, 381, 388, 390, 400, 402, 414, 416, 422, 423, 425, 429, 430, 437, 458, 460]

github-actions[bot] commented 1 month ago

This issue is stale because it has been open for 7 days with no activity.

github-actions[bot] commented 1 month ago

This issue was closed because it has been inactive for 7 days since being marked as stale.