search

filecoin-project / replication-game

Compete on the fastest replication algorithm
http://replication-game.herokuapp.com/
Other
59 stars 8 forks source link

Run failed due to memory allocation #13

Closed whyrusleeping closed 5 years ago

whyrusleeping commented 5 years ago 
why@why-MS-7B46 ~/g/s/g/f/replication-game> ./target/release/replication-game \
                                            --prover $REPL_GAME_ID \
                                            --seed $REPL_GAME_SEED \
                                            --timestamp $REPL_GAME_TIMESTAMP \
                                            --size 1048576 \
                                            zigzag > proof.json
generating fake data
running setup
running replicate
Jan 27 11:47:36.368 INFO encoding, layer {}: 0, place: /home/why/.cargo/git/checkouts/rust-proofs-e06c0682a2ef6285/51df01e/storage-proofs/src/layered_drgporep.rs:33$ storage_proofs::layered_drgporep, root: storage-proofs
Jan 27 11:49:56.632 INFO returning tree, layer: 0, place: /home/why/.cargo/git/checkouts/rust-proofs-e06c0682a2ef6285/51df01e/storage-proofs/src/layered_drgporep.rs$324 storage_proofs::layered_drgporep, root: storage-proofs
Jan 27 11:51:14.603 INFO encoding, layer {}: 1, place: /home/why/.cargo/git/checkouts/rust-proofs-e06c0682a2ef6285/51df01e/storage-proofs/src/layered_drgporep.rs:33$ storage_proofs::layered_drgporep, root: storage-proofs
Jan 27 11:53:34.762 INFO returning tree, layer: 1, place: /home/why/.cargo/git/checkouts/rust-proofs-e06c0682a2ef6285/51df01e/storage-proofs/src/layered_drgporep.rs$324 storage_proofs::layered_drgporep, root: storage-proofs
Jan 27 11:54:53.366 INFO encoding, layer {}: 2, place: /home/why/.cargo/git/checkouts/rust-proofs-e06c0682a2ef6285/51df01e/storage-proofs/src/layered_drgporep.rs:33$ storage_proofs::layered_drgporep, root: storage-proofs

Jan 27 11:57:13.865 INFO returning tree, layer: 2, place: /home/why/.cargo/git/checkouts/rust-proofs-e06c0682a2ef6285/51df01e/storage-proofs/src/layered_drgporep.rs$324 storage_proofs::layered_drgporep, root: storage-proofs
Jan 27 11:58:34.048 INFO encoding, layer {}: 3, place: /home/why/.cargo/git/checkouts/rust-proofs-e06c0682a2ef6285/51df01e/storage-proofs/src/layered_drgporep.rs:33$ storage_proofs::layered_drgporep, root: storage-proofs
Jan 27 12:00:54.267 INFO returning tree, layer: 3, place: /home/why/.cargo/git/checkouts/rust-proofs-e06c0682a2ef6285/51df01e/storage-proofs/src/layered_drgporep.rs$324 storage_proofs::layered_drgporep, root: storage-proofs
Jan 27 12:02:13.071 INFO encoding, layer {}: 4, place: /home/why/.cargo/git/checkouts/rust-proofs-e06c0682a2ef6285/51df01e/storage-proofs/src/layered_drgporep.rs:33$ storage_proofs::layered_drgporep, root: storage-proofs
Jan 27 12:04:33.173 INFO returning tree, layer: 4, place: /home/why/.cargo/git/checkouts/rust-proofs-e06c0682a2ef6285/51df01e/storage-proofs/src/layered_drgporep.rs$324 storage_proofs::layered_drgporep, root: storage-proofs
Jan 27 12:05:53.802 INFO encoding, layer {}: 5, place: /home/why/.cargo/git/checkouts/rust-proofs-e06c0682a2ef6285/51df01e/storage-proofs/src/layered_drgporep.rs:33$ storage_proofs::layered_drgporep, root: storage-proofs
Jan 27 12:08:13.990 INFO returning tree, layer: 5, place: /home/why/.cargo/git/checkouts/rust-proofs-e06c0682a2ef6285/51df01e/storage-proofs/src/layered_drgporep.rs$324 storage_proofs::layered_drgporep, root: storage-proofs
Jan 27 12:09:33.903 INFO encoding, layer {}: 6, place: /home/why/.cargo/git/checkouts/rust-proofs-e06c0682a2ef6285/51df01e/storage-proofs/src/layered_drgporep.rs:331 storage_proofs::layered_drgporep, root: storage-proofs
Jan 27 12:11:53.822 INFO returning tree, layer: 6, place: /home/why/.cargo/git/checkouts/rust-proofs-e06c0682a2ef6285/51df01e/storage-proofs/src/layered_drgporep.rs:324 storage_proofs::layered_drgporep, root: storage-proofs
Jan 27 12:13:15.148 INFO encoding, layer {}: 7, place: /home/why/.cargo/git/checkouts/rust-proofs-e06c0682a2ef6285/51df01e/storage-proofs/src/layered_drgporep.rs:331 storage_proofs::layered_drgporep, root: storage-proofs
Jan 27 12:15:35.356 INFO returning tree, layer: 7, place: /home/why/.cargo/git/checkouts/rust-proofs-e06c0682a2ef6285/51df01e/storage-proofs/src/layered_drgporep.rs:324 storage_proofs::layered_drgporep, root: storage-proofs
Jan 27 12:16:53.818 INFO encoding, layer {}: 8, place: /home/why/.cargo/git/checkouts/rust-proofs-e06c0682a2ef6285/51df01e/storage-proofs/src/layered_drgporep.rs:331 storage_proofs::layered_drgporep, root: storage-proofs
Jan 27 12:19:13.985 INFO returning tree, layer: 8, place: /home/why/.cargo/git/checkouts/rust-proofs-e06c0682a2ef6285/51df01e/storage-proofs/src/layered_drgporep.rs:324 storage_proofs::layered_drgporep, root: storage-proofs
Jan 27 12:20:32.873 INFO encoding, layer {}: 9, place: /home/why/.cargo/git/checkouts/rust-proofs-e06c0682a2ef6285/51df01e/storage-proofs/src/layered_drgporep.rs:331 storage_proofs::layered_drgporep, root: storage-proofs
Jan 27 12:22:53.108 INFO returning tree, layer: 9, place: /home/why/.cargo/git/checkouts/rust-proofs-e06c0682a2ef6285/51df01e/storage-proofs/src/layered_drgporep.rs:324 storage_proofs::layered_drgporep, root: storage-proofs
Jan 27 12:26:23.682 INFO returning tree, layer: 10, place: /home/why/.cargo/git/checkouts/rust-proofs-e06c0682a2ef6285/51df01e/storage-proofs/src/layered_drgporep.rs:324 storage_proofs::layered_drgporep, root: storage-proofs
Jan 27 12:26:24.299 INFO setting tau/aux, layer: 0, place: /home/why/.cargo/git/checkouts/rust-proofs-e06c0682a2ef6285/51df01e/storage-proofs/src/layered_drgporep.rs:370 storage_proofs::layered_drgporep, root: storage-proofs
memory allocation of 2147483616 bytes failedfish: “./target/release/replication-ga…” terminated by signal SIGABRT (Abort)

I have 32 GB of ram, not much else on the system was using it either.

porcuquine commented 5 years ago

For sake of information, I don't hit this problem with 64GB running the zigzag example. (It's possible this is an issue with the replication game only, but probably not.

➜  rust-proofs git:(master) ✗ ./target/release/examples/zigzag --size 1048576 --no-bench
Jan 27 13:11:11.959 INFO hasher: pedersen, target: config, place: filecoin-proofs/examples/zigzag.rs:409 zigzag, root: filecoin-proofs
Jan 27 13:11:11.959 INFO data size: 1 GB, target: config, place: filecoin-proofs/examples/zigzag.rs:107 zigzag, root: filecoin-proofs
Jan 27 13:11:11.960 INFO m: 5, target: config, place: filecoin-proofs/examples/zigzag.rs:108 zigzag, root: filecoin-proofs
Jan 27 13:11:11.960 INFO expansion_degree: 6, target: config, place: filecoin-proofs/examples/zigzag.rs:109 zigzag, root: filecoin-proofs
Jan 27 13:11:11.960 INFO sloth: 0, target: config, place: filecoin-proofs/examples/zigzag.rs:110 zigzag, root: filecoin-proofs
Jan 27 13:11:11.960 INFO challenge_count: 1, target: config, place: filecoin-proofs/examples/zigzag.rs:111 zigzag, root: filecoin-proofs
Jan 27 13:11:11.961 INFO layers: 10, target: config, place: filecoin-proofs/examples/zigzag.rs:112 zigzag, root: filecoin-proofs
Jan 27 13:11:11.961 INFO partitions: 1, target: config, place: filecoin-proofs/examples/zigzag.rs:113 zigzag, root: filecoin-proofs
Jan 27 13:11:11.961 INFO circuit: false, target: config, place: filecoin-proofs/examples/zigzag.rs:114 zigzag, root: filecoin-proofs
Jan 27 13:11:11.962 INFO groth: false, target: config, place: filecoin-proofs/examples/zigzag.rs:115 zigzag, root: filecoin-proofs
Jan 27 13:11:11.962 INFO bench: false, target: config, place: filecoin-proofs/examples/zigzag.rs:116 zigzag, root: filecoin-proofs
Jan 27 13:11:11.962 INFO generating fake data, target: status, place: filecoin-proofs/examples/zigzag.rs:118 zigzag, root: filecoin-proofs
Jan 27 13:11:43.739 INFO running setup, place: filecoin-proofs/examples/zigzag.rs:140 zigzag, root: filecoin-proofs
Jan 27 13:11:43.739 INFO running replicate, place: filecoin-proofs/examples/zigzag.rs:148 zigzag, root: filecoin-proofs
Jan 27 13:11:44.089 INFO encoding, layer {}: 0, place: storage-proofs/src/layered_drgporep.rs:329 storage_proofs::layered_drgporep, root: storage-proofs
Jan 27 13:13:21.303 INFO returning tree, layer: 0, place: storage-proofs/src/layered_drgporep.rs:322 storage_proofs::layered_drgporep, root: storage-proofs
Jan 27 13:15:28.582 INFO encoding, layer {}: 1, place: storage-proofs/src/layered_drgporep.rs:329 storage_proofs::layered_drgporep, root: storage-proofs
Jan 27 13:17:05.551 INFO returning tree, layer: 1, place: storage-proofs/src/layered_drgporep.rs:322 storage_proofs::layered_drgporep, root: storage-proofs
Jan 27 13:19:17.889 INFO encoding, layer {}: 2, place: storage-proofs/src/layered_drgporep.rs:329 storage_proofs::layered_drgporep, root: storage-proofs
Jan 27 13:20:54.970 INFO returning tree, layer: 2, place: storage-proofs/src/layered_drgporep.rs:322 storage_proofs::layered_drgporep, root: storage-proofs
Jan 27 13:23:02.378 INFO encoding, layer {}: 3, place: storage-proofs/src/layered_drgporep.rs:329 storage_proofs::layered_drgporep, root: storage-proofs
Jan 27 13:24:39.539 INFO returning tree, layer: 3, place: storage-proofs/src/layered_drgporep.rs:322 storage_proofs::layered_drgporep, root: storage-proofs
Jan 27 13:26:49.322 INFO encoding, layer {}: 4, place: storage-proofs/src/layered_drgporep.rs:329 storage_proofs::layered_drgporep, root: storage-proofs
Jan 27 13:28:26.424 INFO returning tree, layer: 4, place: storage-proofs/src/layered_drgporep.rs:322 storage_proofs::layered_drgporep, root: storage-proofs
Jan 27 13:30:33.350 INFO encoding, layer {}: 5, place: storage-proofs/src/layered_drgporep.rs:329 storage_proofs::layered_drgporep, root: storage-proofs
Jan 27 13:32:10.527 INFO returning tree, layer: 5, place: storage-proofs/src/layered_drgporep.rs:322 storage_proofs::layered_drgporep, root: storage-proofs
Jan 27 13:34:22.328 INFO encoding, layer {}: 6, place: storage-proofs/src/layered_drgporep.rs:329 storage_proofs::layered_drgporep, root: storage-proofs
Jan 27 13:35:59.877 INFO returning tree, layer: 6, place: storage-proofs/src/layered_drgporep.rs:322 storage_proofs::layered_drgporep, root: storage-proofs
Jan 27 13:38:07.456 INFO encoding, layer {}: 7, place: storage-proofs/src/layered_drgporep.rs:329 storage_proofs::layered_drgporep, root: storage-proofs
Jan 27 13:39:44.528 INFO returning tree, layer: 7, place: storage-proofs/src/layered_drgporep.rs:322 storage_proofs::layered_drgporep, root: storage-proofs
Jan 27 13:41:56.014 INFO encoding, layer {}: 8, place: storage-proofs/src/layered_drgporep.rs:329 storage_proofs::layered_drgporep, root: storage-proofs
Jan 27 13:43:33.155 INFO returning tree, layer: 8, place: storage-proofs/src/layered_drgporep.rs:322 storage_proofs::layered_drgporep, root: storage-proofs
Jan 27 13:45:41.184 INFO encoding, layer {}: 9, place: storage-proofs/src/layered_drgporep.rs:329 storage_proofs::layered_drgporep, root: storage-proofs
Jan 27 13:47:18.315 INFO returning tree, layer: 9, place: storage-proofs/src/layered_drgporep.rs:322 storage_proofs::layered_drgporep, root: storage-proofs
Jan 27 13:51:05.572 INFO returning tree, layer: 10, place: storage-proofs/src/layered_drgporep.rs:322 storage_proofs::layered_drgporep, root: storage-proofs
Jan 27 13:51:06.555 INFO setting tau/aux, layer: 0, place: storage-proofs/src/layered_drgporep.rs:368 storage_proofs::layered_drgporep, root: storage-proofs
Jan 27 13:51:07.504 INFO setting tau/aux, layer: 1, place: storage-proofs/src/layered_drgporep.rs:368 storage_proofs::layered_drgporep, root: storage-proofs
Jan 27 13:51:08.453 INFO setting tau/aux, layer: 2, place: storage-proofs/src/layered_drgporep.rs:368 storage_proofs::layered_drgporep, root: storage-proofs
Jan 27 13:51:09.402 INFO setting tau/aux, layer: 3, place: storage-proofs/src/layered_drgporep.rs:368 storage_proofs::layered_drgporep, root: storage-proofs
Jan 27 13:51:10.352 INFO setting tau/aux, layer: 4, place: storage-proofs/src/layered_drgporep.rs:368 storage_proofs::layered_drgporep, root: storage-proofs
Jan 27 13:51:11.301 INFO setting tau/aux, layer: 5, place: storage-proofs/src/layered_drgporep.rs:368 storage_proofs::layered_drgporep, root: storage-proofs
Jan 27 13:51:12.248 INFO setting tau/aux, layer: 6, place: storage-proofs/src/layered_drgporep.rs:368 storage_proofs::layered_drgporep, root: storage-proofs
Jan 27 13:51:13.194 INFO setting tau/aux, layer: 7, place: storage-proofs/src/layered_drgporep.rs:368 storage_proofs::layered_drgporep, root: storage-proofs
Jan 27 13:51:14.141 INFO setting tau/aux, layer: 8, place: storage-proofs/src/layered_drgporep.rs:368 storage_proofs::layered_drgporep, root: storage-proofs
Jan 27 13:51:15.087 INFO setting tau/aux, layer: 9, place: storage-proofs/src/layered_drgporep.rs:368 storage_proofs::layered_drgporep, root: storage-proofs
Jan 27 13:51:16.495 INFO replication_time: 2372.756497074s, target: stats, place: filecoin-proofs/examples/zigzag.rs:170 zigzag, root: filecoin-proofs
Jan 27 13:51:16.496 INFO replication_time/byte: 2.209µs, target: stats, place: filecoin-proofs/examples/zigzag.rs:171 zigzag, root: filecoin-proofs
Jan 27 13:51:16.496 INFO replication_time/GiB: 2372.756497073s, target: stats, place: filecoin-proofs/examples/zigzag.rs:176 zigzag, root: filecoin-proofs
Jan 27 13:51:16.496 INFO generating one proof, place: filecoin-proofs/examples/zigzag.rs:183 zigzag, root: filecoin-proofs
Jan 27 13:52:38.869 INFO vanilla_proving_time: 82.373391973 seconds, target: stats, place: filecoin-proofs/examples/zigzag.rs:208 zigzag, root: filecoin-proofs
Jan 27 13:52:38.869 INFO sampling verifying (samples: 5), place: filecoin-proofs/examples/zigzag.rs:211 zigzag, root: filecoin-proofs
Jan 27 13:52:39.477 INFO Verification complete, target: status, place: filecoin-proofs/examples/zigzag.rs:225 zigzag, root: filecoin-proofs
Jan 27 13:52:39.477 INFO average_vanilla_verifying_time: 0.121695571 seconds, target: stats, place: filecoin-proofs/examples/zigzag.rs:231 zigzag, root: filecoin-proofs
whyrusleeping commented 5 years ago

Still failed with 64GB of ram: 

Jan 28 18:07:32.322 INFO setting tau/aux, layer: 5, place: /home/why/.cargo/git/checkouts/rust-proofs-e06c0682a2ef6285/51df01e/storage-proofs/src/layered_drgporep.rs:370 storage_proofs::layered_drgporep, root: storage-proofs
Jan 28 18:07:32.871 INFO setting tau/aux, layer: 6, place: /home/why/.cargo/git/checkouts/rust-proofs-e06c0682a2ef6285/51df01e/storage-proofs/src/layered_drgporep.rs:370 storage_proofs::layered_drgporep, root: storage-proofs
Jan 28 18:07:33.423 INFO setting tau/aux, layer: 7, place: /home/why/.cargo/git/checkouts/rust-proofs-e06c0682a2ef6285/51df01e/storage-proofs/src/layered_drgporep.rs:370 storage_proofs::layered_drgporep, root: storage-proofs
Jan 28 18:07:33.974 INFO setting tau/aux, layer: 8, place: /home/why/.cargo/git/checkouts/rust-proofs-e06c0682a2ef6285/51df01e/storage-proofs/src/layered_drgporep.rs:370 storage_proofs::layered_drgporep, root: storage-proofs
Jan 28 18:07:34.526 INFO setting tau/aux, layer: 9, place: /home/why/.cargo/git/checkouts/rust-proofs-e06c0682a2ef6285/51df01e/storage-proofs/src/layered_drgporep.rs:370 storage_proofs::layered_drgporep, root: storage-proofs
generating one proof
memory allocation of 2147483616 bytes failed./game.sh: line 12:  2948 Aborted                 (core dumped) ./target/release/replication-game --prover $REPL_GAME_ID --seed $REPL_GAME_SEED --timestamp $REPL_GAME_TIMESTAMP --size 1048576 zigzag > proof.json
yangjian102621 commented 5 years ago

failed with 16GB ram

./target/release/replication-game \

--prover $REPL_GAME_ID \ --seed $REPL_GAME_SEED \ --timestamp $REPL_GAME_TIMESTAMP \ --size 1048576 \ zigzag > proof.json generating fake data running setup running replicate Mar 05 16:13:14.709 INFO encoding, layer {}: 0, place: /home/yangjian/.cargo/git/checkouts/rust-fil-proofs-4a4e9362af45e2c5/8584f32/storage-proofs/src/layered_drgporep.rs:401 storage_proofs::layered_drgporep, root: storage-proofs Mar 05 16:19:06.134 INFO encoding, layer {}: 1, place: /home/yangjian/.cargo/git/checkouts/rust-fil-proofs-4a4e9362af45e2c5/8584f32/storage-proofs/src/layered_drgporep.rs:401 storage_proofs::layered_drgporep, root: storage-proofs Mar 05 16:29:25.588 INFO encoding, layer {}: 2, place: /home/yangjian/.cargo/git/checkouts/rust-fil-proofs-4a4e9362af45e2c5/8584f32/storage-proofs/src/layered_drgporep.rs:401 storage_proofs::layered_drgporep, root: storage-proofs Mar 05 16:42:55.446 INFO returning tree, layer: 2, place: /home/yangjian/.cargo/git/checkouts/rust-fil-proofs-4a4e9362af45e2c5/8584f32/storage-proofs/src/layered_drgporep.rs:394 storage_proofs::layered_drgporep, root: storage-proofs Mar 05 16:42:55.451 INFO returning tree, layer: 0, place: /home/yangjian/.cargo/git/checkouts/rust-fil-proofs-4a4e9362af45e2c5/8584f32/storage-proofs/src/layered_drgporep.rs:394 storage_proofs::layered_drgporep, root: storage-proofs Mar 05 16:42:55.451 INFO returning tree, layer: 1, place: /home/yangjian/.cargo/git/checkouts/rust-fil-proofs-4a4e9362af45e2c5/8584f32/storage-proofs/src/layered_drgporep.rs:394 storage_proofs::layered_drgporep, root: storage-proofs Mar 05 16:43:43.427 INFO encoding, layer {}: 3, place: /home/yangjian/.cargo/git/checkouts/rust-fil-proofs-4a4e9362af45e2c5/8584f32/storage-proofs/src/layered_drgporep.rs:401 storage_proofs::layered_drgporep, root: storage-proofs Mar 05 16:52:59.663 INFO returning tree, layer: 3, place: /home/yangjian/.cargo/git/checkouts/rust-fil-proofs-4a4e9362af45e2c5/8584f32/storage-proofs/src/layered_drgporep.rs:394 storage_proofs::layered_drgporep, root: storage-proofs Mar 05 16:55:48.125 INFO encoding, layer {}: 4, place: /home/yangjian/.cargo/git/checkouts/rust-fil-proofs-4a4e9362af45e2c5/8584f32/storage-proofs/src/layered_drgporep.rs:401 storage_proofs::layered_drgporep, root: storage-proofs Mar 05 17:05:20.569 INFO returning tree, layer: 4, place: /home/yangjian/.cargo/git/checkouts/rust-fil-proofs-4a4e9362af45e2c5/8584f32/storage-proofs/src/layered_drgporep.rs:394 storage_proofs::layered_drgporep, root: storage-proofs Mar 05 17:07:25.692 INFO encoding, layer {}: 5, place: /home/yangjian/.cargo/git/checkouts/rust-fil-proofs-4a4e9362af45e2c5/8584f32/storage-proofs/src/layered_drgporep.rs:401 storage_proofs::layered_drgporep, root: storage-proofs Mar 05 17:16:50.912 INFO returning tree, layer: 5, place: /home/yangjian/.cargo/git/checkouts/rust-fil-proofs-4a4e9362af45e2c5/8584f32/storage-proofs/src/layered_drgporep.rs:394 storage_proofs::layered_drgporep, root: storage-proofs memory allocation of 2147483616 bytes failedAborted

dignifiedquire commented 5 years ago

new versions require less memory, still a lot though, but that is not an issue with the game itself