search

mo-shahab / vershachi-unlearning

A framework for machine unlearning.
MIT License
2 stars 1 forks source link

sisa: memory error when training the model. #13

Open mo-shahab opened 4 months ago

mo-shahab commented 4 months ago

should probably reduce the number of the requests and the number of epochs, currently, requests=16 and epochs=5 and also, should work on batch loading of the data to the model in the training phase. 

(venv) shahab in C:\dev\vershachi-unlearning\examples on main ● ~1 λ python .\example_sisa_train.py
shard: 1/4, requests: 1/16
Previous checkpoint removed: containers/default/cache/1f99f38a56d49f1a56feff3ea0e749f6c0030e106111e83b5f79a1988da0ac4c_0.pt
Previous time file removed: containers/default/times/1f99f38a56d49f1a56feff3ea0e749f6c0030e106111e83b5f79a1988da0ac4c_0.time
Previous checkpoint removed: containers/default/cache/1f99f38a56d49f1a56feff3ea0e749f6c0030e106111e83b5f79a1988da0ac4c_1.pt
Previous time file removed: containers/default/times/1f99f38a56d49f1a56feff3ea0e749f6c0030e106111e83b5f79a1988da0ac4c_1.time
Previous checkpoint removed: containers/default/cache/1f99f38a56d49f1a56feff3ea0e749f6c0030e106111e83b5f79a1988da0ac4c_2.pt
Previous time file removed: containers/default/times/1f99f38a56d49f1a56feff3ea0e749f6c0030e106111e83b5f79a1988da0ac4c_2.time
Previous checkpoint removed: containers/default/cache/1f99f38a56d49f1a56feff3ea0e749f6c0030e106111e83b5f79a1988da0ac4c_3.pt
Previous time file removed: containers/default/times/1f99f38a56d49f1a56feff3ea0e749f6c0030e106111e83b5f79a1988da0ac4c_3.time
shard: 1/4, requests: 2/16
Previous checkpoint removed: containers/default/cache/1f99f38a56d49f1a56feff3ea0e749f6c0030e106111e83b5f79a1988da0ac4c_0.pt
Previous time file removed: containers/default/times/1f99f38a56d49f1a56feff3ea0e749f6c0030e106111e83b5f79a1988da0ac4c_0.time
Previous checkpoint removed: containers/default/cache/1f99f38a56d49f1a56feff3ea0e749f6c0030e106111e83b5f79a1988da0ac4c_1.pt
Previous time file removed: containers/default/times/1f99f38a56d49f1a56feff3ea0e749f6c0030e106111e83b5f79a1988da0ac4c_1.time
Previous checkpoint removed: containers/default/cache/1f99f38a56d49f1a56feff3ea0e749f6c0030e106111e83b5f79a1988da0ac4c_2.pt
Previous time file removed: containers/default/times/1f99f38a56d49f1a56feff3ea0e749f6c0030e106111e83b5f79a1988da0ac4c_2.time
Previous checkpoint removed: containers/default/cache/1f99f38a56d49f1a56feff3ea0e749f6c0030e106111e83b5f79a1988da0ac4c_3.pt
Previous time file removed: containers/default/times/1f99f38a56d49f1a56feff3ea0e749f6c0030e106111e83b5f79a1988da0ac4c_3.time
shard: 1/4, requests: 3/16
Previous checkpoint removed: containers/default/cache/1f99f38a56d49f1a56feff3ea0e749f6c0030e106111e83b5f79a1988da0ac4c_0.pt
Previous time file removed: containers/default/times/1f99f38a56d49f1a56feff3ea0e749f6c0030e106111e83b5f79a1988da0ac4c_0.time
Previous checkpoint removed: containers/default/cache/1f99f38a56d49f1a56feff3ea0e749f6c0030e106111e83b5f79a1988da0ac4c_1.pt
Previous time file removed: containers/default/times/1f99f38a56d49f1a56feff3ea0e749f6c0030e106111e83b5f79a1988da0ac4c_1.time
Previous checkpoint removed: containers/default/cache/1f99f38a56d49f1a56feff3ea0e749f6c0030e106111e83b5f79a1988da0ac4c_2.pt
Previous time file removed: containers/default/times/1f99f38a56d49f1a56feff3ea0e749f6c0030e106111e83b5f79a1988da0ac4c_2.time
Previous checkpoint removed: containers/default/cache/1f99f38a56d49f1a56feff3ea0e749f6c0030e106111e83b5f79a1988da0ac4c_3.pt
Previous time file removed: containers/default/times/1f99f38a56d49f1a56feff3ea0e749f6c0030e106111e83b5f79a1988da0ac4c_3.time
shard: 1/4, requests: 4/16
Previous checkpoint removed: containers/default/cache/04f1b558d065f8157e14e96376563fffe263bdc76ba1474992fa7cc4b6119785_0.pt
Previous time file removed: containers/default/times/04f1b558d065f8157e14e96376563fffe263bdc76ba1474992fa7cc4b6119785_0.time
Previous checkpoint removed: containers/default/cache/04f1b558d065f8157e14e96376563fffe263bdc76ba1474992fa7cc4b6119785_1.pt
Previous time file removed: containers/default/times/04f1b558d065f8157e14e96376563fffe263bdc76ba1474992fa7cc4b6119785_1.time
Previous checkpoint removed: containers/default/cache/04f1b558d065f8157e14e96376563fffe263bdc76ba1474992fa7cc4b6119785_2.pt
Previous time file removed: containers/default/times/04f1b558d065f8157e14e96376563fffe263bdc76ba1474992fa7cc4b6119785_2.time
Previous checkpoint removed: containers/default/cache/04f1b558d065f8157e14e96376563fffe263bdc76ba1474992fa7cc4b6119785_3.pt
Previous time file removed: containers/default/times/04f1b558d065f8157e14e96376563fffe263bdc76ba1474992fa7cc4b6119785_3.time
shard: 1/4, requests: 5/16
Previous checkpoint removed: containers/default/cache/c0e8001bc8d0bd2801677e8ff143e5a32acfacc0c2da10e0b126bc5ed0c74f17_0.pt
Previous time file removed: containers/default/times/c0e8001bc8d0bd2801677e8ff143e5a32acfacc0c2da10e0b126bc5ed0c74f17_0.time
Previous checkpoint removed: containers/default/cache/c0e8001bc8d0bd2801677e8ff143e5a32acfacc0c2da10e0b126bc5ed0c74f17_1.pt
Previous time file removed: containers/default/times/c0e8001bc8d0bd2801677e8ff143e5a32acfacc0c2da10e0b126bc5ed0c74f17_1.time
Previous checkpoint removed: containers/default/cache/c0e8001bc8d0bd2801677e8ff143e5a32acfacc0c2da10e0b126bc5ed0c74f17_2.pt
Previous time file removed: containers/default/times/c0e8001bc8d0bd2801677e8ff143e5a32acfacc0c2da10e0b126bc5ed0c74f17_2.time
Previous checkpoint removed: containers/default/cache/c0e8001bc8d0bd2801677e8ff143e5a32acfacc0c2da10e0b126bc5ed0c74f17_3.pt
Previous time file removed: containers/default/times/c0e8001bc8d0bd2801677e8ff143e5a32acfacc0c2da10e0b126bc5ed0c74f17_3.time
shard: 1/4, requests: 6/16
Previous checkpoint removed: containers/default/cache/64cff711608dce330c0c5c6bef2aae49fa004cd4c0deeda5351e8b068acde779_0.pt
Previous time file removed: containers/default/times/64cff711608dce330c0c5c6bef2aae49fa004cd4c0deeda5351e8b068acde779_0.time
Previous checkpoint removed: containers/default/cache/64cff711608dce330c0c5c6bef2aae49fa004cd4c0deeda5351e8b068acde779_1.pt
Previous time file removed: containers/default/times/64cff711608dce330c0c5c6bef2aae49fa004cd4c0deeda5351e8b068acde779_1.time
Previous checkpoint removed: containers/default/cache/64cff711608dce330c0c5c6bef2aae49fa004cd4c0deeda5351e8b068acde779_2.pt
Previous time file removed: containers/default/times/64cff711608dce330c0c5c6bef2aae49fa004cd4c0deeda5351e8b068acde779_2.time
Previous checkpoint removed: containers/default/cache/64cff711608dce330c0c5c6bef2aae49fa004cd4c0deeda5351e8b068acde779_3.pt
Previous time file removed: containers/default/times/64cff711608dce330c0c5c6bef2aae49fa004cd4c0deeda5351e8b068acde779_3.time
shard: 1/4, requests: 7/16
Previous checkpoint removed: containers/default/cache/64cff711608dce330c0c5c6bef2aae49fa004cd4c0deeda5351e8b068acde779_0.pt
Previous time file removed: containers/default/times/64cff711608dce330c0c5c6bef2aae49fa004cd4c0deeda5351e8b068acde779_0.time
Previous checkpoint removed: containers/default/cache/64cff711608dce330c0c5c6bef2aae49fa004cd4c0deeda5351e8b068acde779_1.pt
Previous time file removed: containers/default/times/64cff711608dce330c0c5c6bef2aae49fa004cd4c0deeda5351e8b068acde779_1.time
Previous checkpoint removed: containers/default/cache/64cff711608dce330c0c5c6bef2aae49fa004cd4c0deeda5351e8b068acde779_2.pt
Previous time file removed: containers/default/times/64cff711608dce330c0c5c6bef2aae49fa004cd4c0deeda5351e8b068acde779_2.time
Previous checkpoint removed: containers/default/cache/64cff711608dce330c0c5c6bef2aae49fa004cd4c0deeda5351e8b068acde779_3.pt
Previous time file removed: containers/default/times/64cff711608dce330c0c5c6bef2aae49fa004cd4c0deeda5351e8b068acde779_3.time
shard: 1/4, requests: 8/16
Previous checkpoint removed: containers/default/cache/3bbf9e50ba6c917c414068365b856b68755ec38a263909201d6c176446baf6e4_0.pt
Previous time file removed: containers/default/times/3bbf9e50ba6c917c414068365b856b68755ec38a263909201d6c176446baf6e4_0.time
Previous checkpoint removed: containers/default/cache/3bbf9e50ba6c917c414068365b856b68755ec38a263909201d6c176446baf6e4_1.pt
Previous time file removed: containers/default/times/3bbf9e50ba6c917c414068365b856b68755ec38a263909201d6c176446baf6e4_1.time
Previous checkpoint removed: containers/default/cache/3bbf9e50ba6c917c414068365b856b68755ec38a263909201d6c176446baf6e4_2.pt
Previous time file removed: containers/default/times/3bbf9e50ba6c917c414068365b856b68755ec38a263909201d6c176446baf6e4_2.time
Previous checkpoint removed: containers/default/cache/3bbf9e50ba6c917c414068365b856b68755ec38a263909201d6c176446baf6e4_3.pt
Previous time file removed: containers/default/times/3bbf9e50ba6c917c414068365b856b68755ec38a263909201d6c176446baf6e4_3.time
shard: 1/4, requests: 9/16
Previous checkpoint removed: containers/default/cache/5ec3637f24a75b3bb9ba8bd3d997bb4e68ad92d7c1af12711d8c7adbe3bda7a8_0.pt
Previous time file removed: containers/default/times/5ec3637f24a75b3bb9ba8bd3d997bb4e68ad92d7c1af12711d8c7adbe3bda7a8_0.time
Previous checkpoint removed: containers/default/cache/5ec3637f24a75b3bb9ba8bd3d997bb4e68ad92d7c1af12711d8c7adbe3bda7a8_1.pt
Previous time file removed: containers/default/times/5ec3637f24a75b3bb9ba8bd3d997bb4e68ad92d7c1af12711d8c7adbe3bda7a8_1.time
Previous checkpoint removed: containers/default/cache/5ec3637f24a75b3bb9ba8bd3d997bb4e68ad92d7c1af12711d8c7adbe3bda7a8_2.pt
Previous time file removed: containers/default/times/5ec3637f24a75b3bb9ba8bd3d997bb4e68ad92d7c1af12711d8c7adbe3bda7a8_2.time
Previous checkpoint removed: containers/default/cache/5ec3637f24a75b3bb9ba8bd3d997bb4e68ad92d7c1af12711d8c7adbe3bda7a8_3.pt
Previous time file removed: containers/default/times/5ec3637f24a75b3bb9ba8bd3d997bb4e68ad92d7c1af12711d8c7adbe3bda7a8_3.time
shard: 1/4, requests: 10/16
Previous checkpoint removed: containers/default/cache/80a83a27a381c125a1fffa7b280a438e1aea463f1cd4f856cfcb839038ec4f7c_0.pt
Previous time file removed: containers/default/times/80a83a27a381c125a1fffa7b280a438e1aea463f1cd4f856cfcb839038ec4f7c_0.time
Previous checkpoint removed: containers/default/cache/80a83a27a381c125a1fffa7b280a438e1aea463f1cd4f856cfcb839038ec4f7c_1.pt
Previous time file removed: containers/default/times/80a83a27a381c125a1fffa7b280a438e1aea463f1cd4f856cfcb839038ec4f7c_1.time
Previous checkpoint removed: containers/default/cache/80a83a27a381c125a1fffa7b280a438e1aea463f1cd4f856cfcb839038ec4f7c_2.pt
Previous time file removed: containers/default/times/80a83a27a381c125a1fffa7b280a438e1aea463f1cd4f856cfcb839038ec4f7c_2.time
Previous checkpoint removed: containers/default/cache/80a83a27a381c125a1fffa7b280a438e1aea463f1cd4f856cfcb839038ec4f7c_3.pt
Previous time file removed: containers/default/times/80a83a27a381c125a1fffa7b280a438e1aea463f1cd4f856cfcb839038ec4f7c_3.time
shard: 1/4, requests: 11/16
Previous checkpoint removed: containers/default/cache/89d3d5b2d8c01b56c3dab4b6a4874a5bca983c2d2e6ca22c06de4246f7bdd635_0.pt
Previous time file removed: containers/default/times/89d3d5b2d8c01b56c3dab4b6a4874a5bca983c2d2e6ca22c06de4246f7bdd635_0.time
Previous checkpoint removed: containers/default/cache/89d3d5b2d8c01b56c3dab4b6a4874a5bca983c2d2e6ca22c06de4246f7bdd635_1.pt
Previous time file removed: containers/default/times/89d3d5b2d8c01b56c3dab4b6a4874a5bca983c2d2e6ca22c06de4246f7bdd635_1.time
Previous checkpoint removed: containers/default/cache/89d3d5b2d8c01b56c3dab4b6a4874a5bca983c2d2e6ca22c06de4246f7bdd635_2.pt
Previous time file removed: containers/default/times/89d3d5b2d8c01b56c3dab4b6a4874a5bca983c2d2e6ca22c06de4246f7bdd635_2.time
Previous checkpoint removed: containers/default/cache/89d3d5b2d8c01b56c3dab4b6a4874a5bca983c2d2e6ca22c06de4246f7bdd635_3.pt
Previous time file removed: containers/default/times/89d3d5b2d8c01b56c3dab4b6a4874a5bca983c2d2e6ca22c06de4246f7bdd635_3.time
shard: 1/4, requests: 12/16
Previous checkpoint removed: containers/default/cache/89d3d5b2d8c01b56c3dab4b6a4874a5bca983c2d2e6ca22c06de4246f7bdd635_0.pt
Previous time file removed: containers/default/times/89d3d5b2d8c01b56c3dab4b6a4874a5bca983c2d2e6ca22c06de4246f7bdd635_0.time
Previous checkpoint removed: containers/default/cache/89d3d5b2d8c01b56c3dab4b6a4874a5bca983c2d2e6ca22c06de4246f7bdd635_1.pt
Previous time file removed: containers/default/times/89d3d5b2d8c01b56c3dab4b6a4874a5bca983c2d2e6ca22c06de4246f7bdd635_1.time
Previous checkpoint removed: containers/default/cache/89d3d5b2d8c01b56c3dab4b6a4874a5bca983c2d2e6ca22c06de4246f7bdd635_2.pt
Previous time file removed: containers/default/times/89d3d5b2d8c01b56c3dab4b6a4874a5bca983c2d2e6ca22c06de4246f7bdd635_2.time
Previous checkpoint removed: containers/default/cache/89d3d5b2d8c01b56c3dab4b6a4874a5bca983c2d2e6ca22c06de4246f7bdd635_3.pt
Previous time file removed: containers/default/times/89d3d5b2d8c01b56c3dab4b6a4874a5bca983c2d2e6ca22c06de4246f7bdd635_3.time
shard: 1/4, requests: 13/16
Previous checkpoint removed: containers/default/cache/04f1b558d065f8157e14e96376563fffe263bdc76ba1474992fa7cc4b6119785_0.pt
Previous time file removed: containers/default/times/04f1b558d065f8157e14e96376563fffe263bdc76ba1474992fa7cc4b6119785_0.time
Previous checkpoint removed: containers/default/cache/04f1b558d065f8157e14e96376563fffe263bdc76ba1474992fa7cc4b6119785_1.pt
Previous time file removed: containers/default/times/04f1b558d065f8157e14e96376563fffe263bdc76ba1474992fa7cc4b6119785_1.time
Previous checkpoint removed: containers/default/cache/04f1b558d065f8157e14e96376563fffe263bdc76ba1474992fa7cc4b6119785_2.pt
Previous time file removed: containers/default/times/04f1b558d065f8157e14e96376563fffe263bdc76ba1474992fa7cc4b6119785_2.time
Previous checkpoint removed: containers/default/cache/04f1b558d065f8157e14e96376563fffe263bdc76ba1474992fa7cc4b6119785_3.pt
Previous time file removed: containers/default/times/04f1b558d065f8157e14e96376563fffe263bdc76ba1474992fa7cc4b6119785_3.time
shard: 1/4, requests: 14/16
Previous checkpoint removed: containers/default/cache/2b6351e459b58f12ed4a7475a1e5f34017c98daec28a7e2d73e9228ce48cb782_0.pt
Previous time file removed: containers/default/times/2b6351e459b58f12ed4a7475a1e5f34017c98daec28a7e2d73e9228ce48cb782_0.time
Previous checkpoint removed: containers/default/cache/2b6351e459b58f12ed4a7475a1e5f34017c98daec28a7e2d73e9228ce48cb782_1.pt
Previous time file removed: containers/default/times/2b6351e459b58f12ed4a7475a1e5f34017c98daec28a7e2d73e9228ce48cb782_1.time
Previous checkpoint removed: containers/default/cache/2b6351e459b58f12ed4a7475a1e5f34017c98daec28a7e2d73e9228ce48cb782_2.pt
Previous time file removed: containers/default/times/2b6351e459b58f12ed4a7475a1e5f34017c98daec28a7e2d73e9228ce48cb782_2.time
Previous checkpoint removed: containers/default/cache/2b6351e459b58f12ed4a7475a1e5f34017c98daec28a7e2d73e9228ce48cb782_3.pt
Previous time file removed: containers/default/times/2b6351e459b58f12ed4a7475a1e5f34017c98daec28a7e2d73e9228ce48cb782_3.time
shard: 1/4, requests: 15/16
Previous checkpoint removed: containers/default/cache/24ed43ba41b984c79c226499e2b3d1614f419263f6e9d09ed6000c49f011e1a8_0.pt
Previous time file removed: containers/default/times/24ed43ba41b984c79c226499e2b3d1614f419263f6e9d09ed6000c49f011e1a8_0.time
Previous checkpoint removed: containers/default/cache/24ed43ba41b984c79c226499e2b3d1614f419263f6e9d09ed6000c49f011e1a8_1.pt
Previous time file removed: containers/default/times/24ed43ba41b984c79c226499e2b3d1614f419263f6e9d09ed6000c49f011e1a8_1.time
Previous checkpoint removed: containers/default/cache/24ed43ba41b984c79c226499e2b3d1614f419263f6e9d09ed6000c49f011e1a8_2.pt
Previous time file removed: containers/default/times/24ed43ba41b984c79c226499e2b3d1614f419263f6e9d09ed6000c49f011e1a8_2.time
Previous checkpoint removed: containers/default/cache/24ed43ba41b984c79c226499e2b3d1614f419263f6e9d09ed6000c49f011e1a8_3.pt
Previous time file removed: containers/default/times/24ed43ba41b984c79c226499e2b3d1614f419263f6e9d09ed6000c49f011e1a8_3.time
shard: 1/4, requests: 16/16
Previous checkpoint removed: containers/default/cache/45398eb83c6efb42ef0b867b5857d38125f23d57d1b4f55cdc58b0033c0760ba_0.pt
Previous time file removed: containers/default/times/45398eb83c6efb42ef0b867b5857d38125f23d57d1b4f55cdc58b0033c0760ba_0.time
Previous checkpoint removed: containers/default/cache/45398eb83c6efb42ef0b867b5857d38125f23d57d1b4f55cdc58b0033c0760ba_1.pt
Previous time file removed: containers/default/times/45398eb83c6efb42ef0b867b5857d38125f23d57d1b4f55cdc58b0033c0760ba_1.time
Previous checkpoint removed: containers/default/cache/45398eb83c6efb42ef0b867b5857d38125f23d57d1b4f55cdc58b0033c0760ba_2.pt
Previous time file removed: containers/default/times/45398eb83c6efb42ef0b867b5857d38125f23d57d1b4f55cdc58b0033c0760ba_2.time
Previous checkpoint removed: containers/default/cache/45398eb83c6efb42ef0b867b5857d38125f23d57d1b4f55cdc58b0033c0760ba_3.pt
Previous time file removed: containers/default/times/45398eb83c6efb42ef0b867b5857d38125f23d57d1b4f55cdc58b0033c0760ba_3.time
shard: 2/4, requests: 1/16
Previous checkpoint removed: containers/default/cache/fea2ec41a7ba4d3afa03c015d949b82be37fbf7dc0ab3c36a9533dd3a31847ee_0.pt
Previous time file removed: containers/default/times/fea2ec41a7ba4d3afa03c015d949b82be37fbf7dc0ab3c36a9533dd3a31847ee_0.time
Previous checkpoint removed: containers/default/cache/fea2ec41a7ba4d3afa03c015d949b82be37fbf7dc0ab3c36a9533dd3a31847ee_1.pt
Previous time file removed: containers/default/times/fea2ec41a7ba4d3afa03c015d949b82be37fbf7dc0ab3c36a9533dd3a31847ee_1.time
Previous checkpoint removed: containers/default/cache/fea2ec41a7ba4d3afa03c015d949b82be37fbf7dc0ab3c36a9533dd3a31847ee_2.pt
Previous time file removed: containers/default/times/fea2ec41a7ba4d3afa03c015d949b82be37fbf7dc0ab3c36a9533dd3a31847ee_2.time
Previous checkpoint removed: containers/default/cache/fea2ec41a7ba4d3afa03c015d949b82be37fbf7dc0ab3c36a9533dd3a31847ee_3.pt
Previous time file removed: containers/default/times/fea2ec41a7ba4d3afa03c015d949b82be37fbf7dc0ab3c36a9533dd3a31847ee_3.time
shard: 2/4, requests: 2/16
Previous checkpoint removed: containers/default/cache/fea2ec41a7ba4d3afa03c015d949b82be37fbf7dc0ab3c36a9533dd3a31847ee_0.pt
Previous time file removed: containers/default/times/fea2ec41a7ba4d3afa03c015d949b82be37fbf7dc0ab3c36a9533dd3a31847ee_0.time
Previous checkpoint removed: containers/default/cache/fea2ec41a7ba4d3afa03c015d949b82be37fbf7dc0ab3c36a9533dd3a31847ee_1.pt
Previous time file removed: containers/default/times/fea2ec41a7ba4d3afa03c015d949b82be37fbf7dc0ab3c36a9533dd3a31847ee_1.time
Previous checkpoint removed: containers/default/cache/fea2ec41a7ba4d3afa03c015d949b82be37fbf7dc0ab3c36a9533dd3a31847ee_2.pt
Previous time file removed: containers/default/times/fea2ec41a7ba4d3afa03c015d949b82be37fbf7dc0ab3c36a9533dd3a31847ee_2.time
Previous checkpoint removed: containers/default/cache/fea2ec41a7ba4d3afa03c015d949b82be37fbf7dc0ab3c36a9533dd3a31847ee_3.pt
Previous time file removed: containers/default/times/fea2ec41a7ba4d3afa03c015d949b82be37fbf7dc0ab3c36a9533dd3a31847ee_3.time
shard: 2/4, requests: 3/16
Previous checkpoint removed: containers/default/cache/fea2ec41a7ba4d3afa03c015d949b82be37fbf7dc0ab3c36a9533dd3a31847ee_0.pt
Previous time file removed: containers/default/times/fea2ec41a7ba4d3afa03c015d949b82be37fbf7dc0ab3c36a9533dd3a31847ee_0.time
Previous checkpoint removed: containers/default/cache/fea2ec41a7ba4d3afa03c015d949b82be37fbf7dc0ab3c36a9533dd3a31847ee_1.pt
Previous time file removed: containers/default/times/fea2ec41a7ba4d3afa03c015d949b82be37fbf7dc0ab3c36a9533dd3a31847ee_1.time
Previous checkpoint removed: containers/default/cache/fea2ec41a7ba4d3afa03c015d949b82be37fbf7dc0ab3c36a9533dd3a31847ee_2.pt
Previous time file removed: containers/default/times/fea2ec41a7ba4d3afa03c015d949b82be37fbf7dc0ab3c36a9533dd3a31847ee_2.time
Previous checkpoint removed: containers/default/cache/fea2ec41a7ba4d3afa03c015d949b82be37fbf7dc0ab3c36a9533dd3a31847ee_3.pt
Previous time file removed: containers/default/times/fea2ec41a7ba4d3afa03c015d949b82be37fbf7dc0ab3c36a9533dd3a31847ee_3.time
shard: 2/4, requests: 4/16
Previous checkpoint removed: containers/default/cache/fea2ec41a7ba4d3afa03c015d949b82be37fbf7dc0ab3c36a9533dd3a31847ee_0.pt
Previous time file removed: containers/default/times/fea2ec41a7ba4d3afa03c015d949b82be37fbf7dc0ab3c36a9533dd3a31847ee_0.time
Previous checkpoint removed: containers/default/cache/fea2ec41a7ba4d3afa03c015d949b82be37fbf7dc0ab3c36a9533dd3a31847ee_1.pt
Previous time file removed: containers/default/times/fea2ec41a7ba4d3afa03c015d949b82be37fbf7dc0ab3c36a9533dd3a31847ee_1.time
Previous checkpoint removed: containers/default/cache/fea2ec41a7ba4d3afa03c015d949b82be37fbf7dc0ab3c36a9533dd3a31847ee_2.pt
Previous time file removed: containers/default/times/fea2ec41a7ba4d3afa03c015d949b82be37fbf7dc0ab3c36a9533dd3a31847ee_2.time
Previous checkpoint removed: containers/default/cache/fea2ec41a7ba4d3afa03c015d949b82be37fbf7dc0ab3c36a9533dd3a31847ee_3.pt
Previous time file removed: containers/default/times/fea2ec41a7ba4d3afa03c015d949b82be37fbf7dc0ab3c36a9533dd3a31847ee_3.time
shard: 2/4, requests: 5/16
Previous checkpoint removed: containers/default/cache/fea2ec41a7ba4d3afa03c015d949b82be37fbf7dc0ab3c36a9533dd3a31847ee_0.pt
Previous time file removed: containers/default/times/fea2ec41a7ba4d3afa03c015d949b82be37fbf7dc0ab3c36a9533dd3a31847ee_0.time
Previous checkpoint removed: containers/default/cache/fea2ec41a7ba4d3afa03c015d949b82be37fbf7dc0ab3c36a9533dd3a31847ee_1.pt
Previous time file removed: containers/default/times/fea2ec41a7ba4d3afa03c015d949b82be37fbf7dc0ab3c36a9533dd3a31847ee_1.time
Previous checkpoint removed: containers/default/cache/fea2ec41a7ba4d3afa03c015d949b82be37fbf7dc0ab3c36a9533dd3a31847ee_2.pt
Previous time file removed: containers/default/times/fea2ec41a7ba4d3afa03c015d949b82be37fbf7dc0ab3c36a9533dd3a31847ee_2.time
Previous checkpoint removed: containers/default/cache/fea2ec41a7ba4d3afa03c015d949b82be37fbf7dc0ab3c36a9533dd3a31847ee_3.pt
Previous time file removed: containers/default/times/fea2ec41a7ba4d3afa03c015d949b82be37fbf7dc0ab3c36a9533dd3a31847ee_3.time
shard: 2/4, requests: 6/16
Previous checkpoint removed: containers/default/cache/fa0663712f0ac64b13f3b5f402e229b3348021ea41b01a96f18f18626f80bf22_0.pt
Previous time file removed: containers/default/times/fa0663712f0ac64b13f3b5f402e229b3348021ea41b01a96f18f18626f80bf22_0.time
Previous checkpoint removed: containers/default/cache/fa0663712f0ac64b13f3b5f402e229b3348021ea41b01a96f18f18626f80bf22_1.pt
Previous time file removed: containers/default/times/fa0663712f0ac64b13f3b5f402e229b3348021ea41b01a96f18f18626f80bf22_1.time
Traceback (most recent call last):
  File "C:\dev\vershachi-unlearning\examples\example_sisa_train.py", line 23, in <module>
    trainer._train()
  File "C:\dev\vershachi-unlearning\vershachi\sisa\sisa.py", line 159, in _train
    for images, labels in fetchShardBatch(
  File "C:\dev\vershachi-unlearning\vershachi\sisa\sharded.py", line 96, in fetchShardBatch
    spec.loader.exec_module(dataloader_module)
  File "<frozen importlib._bootstrap_external>", line 994, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "C:\dev\vershachi-unlearning\datasets\dataloader.py", line 6, in <module>
    train_data = np.load(train_data_path, allow_pickle=True).item()
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\dev\vershachi-unlearning\venv\Lib\site-packages\numpy\lib\npyio.py", line 456, in load
    return format.read_array(fid, allow_pickle=allow_pickle,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\dev\vershachi-unlearning\venv\Lib\site-packages\numpy\lib\format.py", line 800, in read_array
    array = pickle.load(fp, **pickle_kwargs)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
MemoryError
mo-shahab commented 4 months ago

after reducing the number of requests and running the program without anything open (to give as much as memory possible for the execution) it slightly works but, this is encountered

(venv) shahab in C:\dev\vershachi-unlearning\examples on main ● ~1 λ python .\example_sisa_train.py
shard: 1/4, requests: 1/5
Previous checkpoint removed: containers/default/cache/1f99f38a56d49f1a56feff3ea0e749f6c0030e106111e83b5f79a1988da0ac4c_0.pt
Previous time file removed: containers/default/times/1f99f38a56d49f1a56feff3ea0e749f6c0030e106111e83b5f79a1988da0ac4c_0.time
Previous checkpoint removed: containers/default/cache/1f99f38a56d49f1a56feff3ea0e749f6c0030e106111e83b5f79a1988da0ac4c_1.pt
Previous time file removed: containers/default/times/1f99f38a56d49f1a56feff3ea0e749f6c0030e106111e83b5f79a1988da0ac4c_1.time
Previous checkpoint removed: containers/default/cache/1f99f38a56d49f1a56feff3ea0e749f6c0030e106111e83b5f79a1988da0ac4c_2.pt
Previous time file removed: containers/default/times/1f99f38a56d49f1a56feff3ea0e749f6c0030e106111e83b5f79a1988da0ac4c_2.time
Previous checkpoint removed: containers/default/cache/1f99f38a56d49f1a56feff3ea0e749f6c0030e106111e83b5f79a1988da0ac4c_3.pt
Previous time file removed: containers/default/times/1f99f38a56d49f1a56feff3ea0e749f6c0030e106111e83b5f79a1988da0ac4c_3.time
shard: 1/4, requests: 2/5
Previous checkpoint removed: containers/default/cache/1f99f38a56d49f1a56feff3ea0e749f6c0030e106111e83b5f79a1988da0ac4c_0.pt
Previous time file removed: containers/default/times/1f99f38a56d49f1a56feff3ea0e749f6c0030e106111e83b5f79a1988da0ac4c_0.time
Previous checkpoint removed: containers/default/cache/1f99f38a56d49f1a56feff3ea0e749f6c0030e106111e83b5f79a1988da0ac4c_1.pt
Previous time file removed: containers/default/times/1f99f38a56d49f1a56feff3ea0e749f6c0030e106111e83b5f79a1988da0ac4c_1.time
Previous checkpoint removed: containers/default/cache/1f99f38a56d49f1a56feff3ea0e749f6c0030e106111e83b5f79a1988da0ac4c_2.pt
Previous time file removed: containers/default/times/1f99f38a56d49f1a56feff3ea0e749f6c0030e106111e83b5f79a1988da0ac4c_2.time
Previous checkpoint removed: containers/default/cache/1f99f38a56d49f1a56feff3ea0e749f6c0030e106111e83b5f79a1988da0ac4c_3.pt
Previous time file removed: containers/default/times/1f99f38a56d49f1a56feff3ea0e749f6c0030e106111e83b5f79a1988da0ac4c_3.time
shard: 1/4, requests: 3/5
Previous checkpoint removed: containers/default/cache/1f99f38a56d49f1a56feff3ea0e749f6c0030e106111e83b5f79a1988da0ac4c_0.pt
Previous time file removed: containers/default/times/1f99f38a56d49f1a56feff3ea0e749f6c0030e106111e83b5f79a1988da0ac4c_0.time
Previous checkpoint removed: containers/default/cache/1f99f38a56d49f1a56feff3ea0e749f6c0030e106111e83b5f79a1988da0ac4c_1.pt
Previous time file removed: containers/default/times/1f99f38a56d49f1a56feff3ea0e749f6c0030e106111e83b5f79a1988da0ac4c_1.time
Previous checkpoint removed: containers/default/cache/1f99f38a56d49f1a56feff3ea0e749f6c0030e106111e83b5f79a1988da0ac4c_2.pt
Previous time file removed: containers/default/times/1f99f38a56d49f1a56feff3ea0e749f6c0030e106111e83b5f79a1988da0ac4c_2.time
Previous checkpoint removed: containers/default/cache/1f99f38a56d49f1a56feff3ea0e749f6c0030e106111e83b5f79a1988da0ac4c_3.pt
Previous time file removed: containers/default/times/1f99f38a56d49f1a56feff3ea0e749f6c0030e106111e83b5f79a1988da0ac4c_3.time
shard: 1/4, requests: 4/5
Previous checkpoint removed: containers/default/cache/04f1b558d065f8157e14e96376563fffe263bdc76ba1474992fa7cc4b6119785_0.pt
Previous time file removed: containers/default/times/04f1b558d065f8157e14e96376563fffe263bdc76ba1474992fa7cc4b6119785_0.time
Previous checkpoint removed: containers/default/cache/04f1b558d065f8157e14e96376563fffe263bdc76ba1474992fa7cc4b6119785_1.pt
Previous time file removed: containers/default/times/04f1b558d065f8157e14e96376563fffe263bdc76ba1474992fa7cc4b6119785_1.time
Previous checkpoint removed: containers/default/cache/04f1b558d065f8157e14e96376563fffe263bdc76ba1474992fa7cc4b6119785_2.pt
Previous time file removed: containers/default/times/04f1b558d065f8157e14e96376563fffe263bdc76ba1474992fa7cc4b6119785_2.time
Previous checkpoint removed: containers/default/cache/04f1b558d065f8157e14e96376563fffe263bdc76ba1474992fa7cc4b6119785_3.pt
Previous time file removed: containers/default/times/04f1b558d065f8157e14e96376563fffe263bdc76ba1474992fa7cc4b6119785_3.time
shard: 1/4, requests: 5/5
Previous checkpoint removed: containers/default/cache/c0e8001bc8d0bd2801677e8ff143e5a32acfacc0c2da10e0b126bc5ed0c74f17_0.pt
Previous time file removed: containers/default/times/c0e8001bc8d0bd2801677e8ff143e5a32acfacc0c2da10e0b126bc5ed0c74f17_0.time
Previous checkpoint removed: containers/default/cache/c0e8001bc8d0bd2801677e8ff143e5a32acfacc0c2da10e0b126bc5ed0c74f17_1.pt
Previous time file removed: containers/default/times/c0e8001bc8d0bd2801677e8ff143e5a32acfacc0c2da10e0b126bc5ed0c74f17_1.time
Previous checkpoint removed: containers/default/cache/c0e8001bc8d0bd2801677e8ff143e5a32acfacc0c2da10e0b126bc5ed0c74f17_2.pt
Previous time file removed: containers/default/times/c0e8001bc8d0bd2801677e8ff143e5a32acfacc0c2da10e0b126bc5ed0c74f17_2.time
Previous checkpoint removed: containers/default/cache/c0e8001bc8d0bd2801677e8ff143e5a32acfacc0c2da10e0b126bc5ed0c74f17_3.pt
Previous time file removed: containers/default/times/c0e8001bc8d0bd2801677e8ff143e5a32acfacc0c2da10e0b126bc5ed0c74f17_3.time
shard: 2/4, requests: 1/5
Previous checkpoint removed: containers/default/cache/fea2ec41a7ba4d3afa03c015d949b82be37fbf7dc0ab3c36a9533dd3a31847ee_0.pt
Previous time file removed: containers/default/times/fea2ec41a7ba4d3afa03c015d949b82be37fbf7dc0ab3c36a9533dd3a31847ee_0.time
Previous checkpoint removed: containers/default/cache/fea2ec41a7ba4d3afa03c015d949b82be37fbf7dc0ab3c36a9533dd3a31847ee_1.pt
Previous time file removed: containers/default/times/fea2ec41a7ba4d3afa03c015d949b82be37fbf7dc0ab3c36a9533dd3a31847ee_1.time
Previous checkpoint removed: containers/default/cache/fea2ec41a7ba4d3afa03c015d949b82be37fbf7dc0ab3c36a9533dd3a31847ee_2.pt
Previous time file removed: containers/default/times/fea2ec41a7ba4d3afa03c015d949b82be37fbf7dc0ab3c36a9533dd3a31847ee_2.time
Previous checkpoint removed: containers/default/cache/fea2ec41a7ba4d3afa03c015d949b82be37fbf7dc0ab3c36a9533dd3a31847ee_3.pt
Previous time file removed: containers/default/times/fea2ec41a7ba4d3afa03c015d949b82be37fbf7dc0ab3c36a9533dd3a31847ee_3.time
shard: 2/4, requests: 2/5
Previous checkpoint removed: containers/default/cache/fea2ec41a7ba4d3afa03c015d949b82be37fbf7dc0ab3c36a9533dd3a31847ee_0.pt
Previous time file removed: containers/default/times/fea2ec41a7ba4d3afa03c015d949b82be37fbf7dc0ab3c36a9533dd3a31847ee_0.time
Previous checkpoint removed: containers/default/cache/fea2ec41a7ba4d3afa03c015d949b82be37fbf7dc0ab3c36a9533dd3a31847ee_1.pt
Previous time file removed: containers/default/times/fea2ec41a7ba4d3afa03c015d949b82be37fbf7dc0ab3c36a9533dd3a31847ee_1.time
Previous checkpoint removed: containers/default/cache/fea2ec41a7ba4d3afa03c015d949b82be37fbf7dc0ab3c36a9533dd3a31847ee_2.pt
Previous time file removed: containers/default/times/fea2ec41a7ba4d3afa03c015d949b82be37fbf7dc0ab3c36a9533dd3a31847ee_2.time
Previous checkpoint removed: containers/default/cache/fea2ec41a7ba4d3afa03c015d949b82be37fbf7dc0ab3c36a9533dd3a31847ee_3.pt
Previous time file removed: containers/default/times/fea2ec41a7ba4d3afa03c015d949b82be37fbf7dc0ab3c36a9533dd3a31847ee_3.time
shard: 2/4, requests: 3/5
Previous checkpoint removed: containers/default/cache/fea2ec41a7ba4d3afa03c015d949b82be37fbf7dc0ab3c36a9533dd3a31847ee_0.pt
Previous time file removed: containers/default/times/fea2ec41a7ba4d3afa03c015d949b82be37fbf7dc0ab3c36a9533dd3a31847ee_0.time
Previous checkpoint removed: containers/default/cache/fea2ec41a7ba4d3afa03c015d949b82be37fbf7dc0ab3c36a9533dd3a31847ee_1.pt
Previous time file removed: containers/default/times/fea2ec41a7ba4d3afa03c015d949b82be37fbf7dc0ab3c36a9533dd3a31847ee_1.time
Previous checkpoint removed: containers/default/cache/fea2ec41a7ba4d3afa03c015d949b82be37fbf7dc0ab3c36a9533dd3a31847ee_2.pt
Previous time file removed: containers/default/times/fea2ec41a7ba4d3afa03c015d949b82be37fbf7dc0ab3c36a9533dd3a31847ee_2.time
Previous checkpoint removed: containers/default/cache/fea2ec41a7ba4d3afa03c015d949b82be37fbf7dc0ab3c36a9533dd3a31847ee_3.pt
Previous time file removed: containers/default/times/fea2ec41a7ba4d3afa03c015d949b82be37fbf7dc0ab3c36a9533dd3a31847ee_3.time
shard: 2/4, requests: 4/5
Previous checkpoint removed: containers/default/cache/fea2ec41a7ba4d3afa03c015d949b82be37fbf7dc0ab3c36a9533dd3a31847ee_0.pt
Previous time file removed: containers/default/times/fea2ec41a7ba4d3afa03c015d949b82be37fbf7dc0ab3c36a9533dd3a31847ee_0.time
Previous checkpoint removed: containers/default/cache/fea2ec41a7ba4d3afa03c015d949b82be37fbf7dc0ab3c36a9533dd3a31847ee_1.pt
Previous time file removed: containers/default/times/fea2ec41a7ba4d3afa03c015d949b82be37fbf7dc0ab3c36a9533dd3a31847ee_1.time
Previous checkpoint removed: containers/default/cache/fea2ec41a7ba4d3afa03c015d949b82be37fbf7dc0ab3c36a9533dd3a31847ee_2.pt
Previous time file removed: containers/default/times/fea2ec41a7ba4d3afa03c015d949b82be37fbf7dc0ab3c36a9533dd3a31847ee_2.time
Previous checkpoint removed: containers/default/cache/fea2ec41a7ba4d3afa03c015d949b82be37fbf7dc0ab3c36a9533dd3a31847ee_3.pt
Previous time file removed: containers/default/times/fea2ec41a7ba4d3afa03c015d949b82be37fbf7dc0ab3c36a9533dd3a31847ee_3.time
shard: 2/4, requests: 5/5
Previous checkpoint removed: containers/default/cache/fea2ec41a7ba4d3afa03c015d949b82be37fbf7dc0ab3c36a9533dd3a31847ee_0.pt
Previous time file removed: containers/default/times/fea2ec41a7ba4d3afa03c015d949b82be37fbf7dc0ab3c36a9533dd3a31847ee_0.time
Previous checkpoint removed: containers/default/cache/fea2ec41a7ba4d3afa03c015d949b82be37fbf7dc0ab3c36a9533dd3a31847ee_1.pt
Previous time file removed: containers/default/times/fea2ec41a7ba4d3afa03c015d949b82be37fbf7dc0ab3c36a9533dd3a31847ee_1.time
Previous checkpoint removed: containers/default/cache/fea2ec41a7ba4d3afa03c015d949b82be37fbf7dc0ab3c36a9533dd3a31847ee_2.pt
Previous time file removed: containers/default/times/fea2ec41a7ba4d3afa03c015d949b82be37fbf7dc0ab3c36a9533dd3a31847ee_2.time
Previous checkpoint removed: containers/default/cache/fea2ec41a7ba4d3afa03c015d949b82be37fbf7dc0ab3c36a9533dd3a31847ee_3.pt
Previous time file removed: containers/default/times/fea2ec41a7ba4d3afa03c015d949b82be37fbf7dc0ab3c36a9533dd3a31847ee_3.time
shard: 3/4, requests: 1/5
Previous checkpoint removed: containers/default/cache/45812b7774ae4b83f9ba74fdef06750da70440e09c3be34412c21b0eabeedc5f_0.pt
Previous time file removed: containers/default/times/45812b7774ae4b83f9ba74fdef06750da70440e09c3be34412c21b0eabeedc5f_0.time
Previous checkpoint removed: containers/default/cache/45812b7774ae4b83f9ba74fdef06750da70440e09c3be34412c21b0eabeedc5f_1.pt
Previous time file removed: containers/default/times/45812b7774ae4b83f9ba74fdef06750da70440e09c3be34412c21b0eabeedc5f_1.time
Previous checkpoint removed: containers/default/cache/45812b7774ae4b83f9ba74fdef06750da70440e09c3be34412c21b0eabeedc5f_2.pt
Previous time file removed: containers/default/times/45812b7774ae4b83f9ba74fdef06750da70440e09c3be34412c21b0eabeedc5f_2.time
Previous checkpoint removed: containers/default/cache/45812b7774ae4b83f9ba74fdef06750da70440e09c3be34412c21b0eabeedc5f_3.pt
Previous time file removed: containers/default/times/45812b7774ae4b83f9ba74fdef06750da70440e09c3be34412c21b0eabeedc5f_3.time
shard: 3/4, requests: 2/5
Previous checkpoint removed: containers/default/cache/45812b7774ae4b83f9ba74fdef06750da70440e09c3be34412c21b0eabeedc5f_0.pt
Previous time file removed: containers/default/times/45812b7774ae4b83f9ba74fdef06750da70440e09c3be34412c21b0eabeedc5f_0.time
Previous checkpoint removed: containers/default/cache/45812b7774ae4b83f9ba74fdef06750da70440e09c3be34412c21b0eabeedc5f_1.pt
Previous time file removed: containers/default/times/45812b7774ae4b83f9ba74fdef06750da70440e09c3be34412c21b0eabeedc5f_1.time
Previous checkpoint removed: containers/default/cache/45812b7774ae4b83f9ba74fdef06750da70440e09c3be34412c21b0eabeedc5f_2.pt
Previous time file removed: containers/default/times/45812b7774ae4b83f9ba74fdef06750da70440e09c3be34412c21b0eabeedc5f_2.time
Previous checkpoint removed: containers/default/cache/45812b7774ae4b83f9ba74fdef06750da70440e09c3be34412c21b0eabeedc5f_3.pt
Previous time file removed: containers/default/times/45812b7774ae4b83f9ba74fdef06750da70440e09c3be34412c21b0eabeedc5f_3.time
shard: 3/4, requests: 3/5
Previous checkpoint removed: containers/default/cache/45812b7774ae4b83f9ba74fdef06750da70440e09c3be34412c21b0eabeedc5f_0.pt
Previous time file removed: containers/default/times/45812b7774ae4b83f9ba74fdef06750da70440e09c3be34412c21b0eabeedc5f_0.time
Previous checkpoint removed: containers/default/cache/45812b7774ae4b83f9ba74fdef06750da70440e09c3be34412c21b0eabeedc5f_1.pt
Previous time file removed: containers/default/times/45812b7774ae4b83f9ba74fdef06750da70440e09c3be34412c21b0eabeedc5f_1.time
Previous checkpoint removed: containers/default/cache/45812b7774ae4b83f9ba74fdef06750da70440e09c3be34412c21b0eabeedc5f_2.pt
Previous time file removed: containers/default/times/45812b7774ae4b83f9ba74fdef06750da70440e09c3be34412c21b0eabeedc5f_2.time
Previous checkpoint removed: containers/default/cache/45812b7774ae4b83f9ba74fdef06750da70440e09c3be34412c21b0eabeedc5f_3.pt
Previous time file removed: containers/default/times/45812b7774ae4b83f9ba74fdef06750da70440e09c3be34412c21b0eabeedc5f_3.time
shard: 3/4, requests: 4/5
Previous checkpoint removed: containers/default/cache/45812b7774ae4b83f9ba74fdef06750da70440e09c3be34412c21b0eabeedc5f_0.pt
Previous time file removed: containers/default/times/45812b7774ae4b83f9ba74fdef06750da70440e09c3be34412c21b0eabeedc5f_0.time
Previous checkpoint removed: containers/default/cache/45812b7774ae4b83f9ba74fdef06750da70440e09c3be34412c21b0eabeedc5f_1.pt
Previous time file removed: containers/default/times/45812b7774ae4b83f9ba74fdef06750da70440e09c3be34412c21b0eabeedc5f_1.time
Previous checkpoint removed: containers/default/cache/45812b7774ae4b83f9ba74fdef06750da70440e09c3be34412c21b0eabeedc5f_2.pt
Previous time file removed: containers/default/times/45812b7774ae4b83f9ba74fdef06750da70440e09c3be34412c21b0eabeedc5f_2.time
Previous checkpoint removed: containers/default/cache/45812b7774ae4b83f9ba74fdef06750da70440e09c3be34412c21b0eabeedc5f_3.pt
Previous time file removed: containers/default/times/45812b7774ae4b83f9ba74fdef06750da70440e09c3be34412c21b0eabeedc5f_3.time
shard: 3/4, requests: 5/5
Previous checkpoint removed: containers/default/cache/45812b7774ae4b83f9ba74fdef06750da70440e09c3be34412c21b0eabeedc5f_0.pt
Previous time file removed: containers/default/times/45812b7774ae4b83f9ba74fdef06750da70440e09c3be34412c21b0eabeedc5f_0.time
Previous checkpoint removed: containers/default/cache/45812b7774ae4b83f9ba74fdef06750da70440e09c3be34412c21b0eabeedc5f_1.pt
Previous time file removed: containers/default/times/45812b7774ae4b83f9ba74fdef06750da70440e09c3be34412c21b0eabeedc5f_1.time
Previous checkpoint removed: containers/default/cache/45812b7774ae4b83f9ba74fdef06750da70440e09c3be34412c21b0eabeedc5f_2.pt
Previous time file removed: containers/default/times/45812b7774ae4b83f9ba74fdef06750da70440e09c3be34412c21b0eabeedc5f_2.time
Previous checkpoint removed: containers/default/cache/45812b7774ae4b83f9ba74fdef06750da70440e09c3be34412c21b0eabeedc5f_3.pt
Previous time file removed: containers/default/times/45812b7774ae4b83f9ba74fdef06750da70440e09c3be34412c21b0eabeedc5f_3.time
shard: 4/4, requests: 1/5
Traceback (most recent call last):
  File "C:\dev\vershachi-unlearning\examples\example_sisa_train.py", line 27, in <module>
    trainer._train()
  File "C:\dev\vershachi-unlearning\vershachi\sisa\sisa.py", line 159, in _train
    for images, labels in fetchShardBatch(
  File "C:\dev\vershachi-unlearning\vershachi\sisa\sharded.py", line 107, in fetchShardBatch
    yield dataloader_module.load(indices)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\dev\vershachi-unlearning\datasets\dataloader.py", line 21, in load
    return X_train[indices], y_train[indices]
           ~~~~~~~^^^^^^^^^
IndexError: index 249215 is out of bounds for axis 0 with size 249215

it says how index is out of range, it does generate the .pt checkpoint files, which are the checkpoints created by pytorch. it can be viewed as the model being trained, also reduced the number of epochs which slightly may cause a problem in future of the development because of insufficient training of the model