Closed nivkner closed 5 years ago
We do have a memory leak detector for leaks in the interpreted code. It only works on linux, but you should be getting an error if memory was leaked.
If it's a memory leak in miri, it might be related to something other than the actual interpretation. Is the leak amount related to the size of the json array?
i ran the program with multiple configurations, here are the results:
value | size (number of times the value repeats in buffer) | memory (in Gigabytes) |
---|---|---|
0 | 1000 | 0.3 |
0 | 1500 | 0.6 |
0 | 2000 | 1.0 |
1500 | 1000 | 0.8 |
1500 | 1500 | 1.8 |
1500 | 2000 | 3.2 |
edit: changed column from repetitions
to size
and clarified
what is value
and repetitions
?
i put a 1000 times the value 0 in the buffer for example
Is this max-RSS, or how do you measure memory consumption?
this problem was originally found with a program that doesn't use serde, but this is the smallest example I found.
If you had an example (even a bigger one) that does not depend on any other crates, that would be helpful.
Interpreting any serde related code could end up using a lot of memory because we're loading all the rlibs and metadata of those crates into miri in order to be able to evalute the MIR of the serde code. Why that would be affected by the size of the json is not clear to me.
i pushed a version of the program that doesn't use serde
here (it uses libc
but only for the type aliases).
i measured the memory by monitoring the total memory usage, so for example:
with the browser closed i am on a stable 1.5 Gigabytes. when i run the program it rises to say 3.8 Gigabytes before the program is done, so the diff is what i write.
note that this is for measuring the serde
example. the program i linked to uses a much bigger buffer (20736) so you might have to prepare to kill the process.
So... I'm assuming you have 4GB of RAM? Your system getting stuck is probably due to your system starting to offload any additional memory to disk (swap).
Do note that miri is a few orders of magnitude slower than executing your program directly. Also it uses a few multiples of the amount of memory that your program uses in debug mode. You might simply be seeing the overhead.
I suggest to run miri with small buffers and few repetitions. Even if that isn't what your real program does, it should still show most of the same behavior.
@oli-obk i pushed a commit to the example that reuses the same stack-allocated buffer throughout the entire loop, the memory usage keeps growing like before.
I measured the memory usage like before, i killed the process when the system memory usage was at 7.6 (I have a 8 GB system), but it was still growing.
I'll do some investigating, but it might just be expected, especially with the stacked borrows impl that the max memory usage explodes, because every time you take an address a few elements are pushed onto some stacks. Even if those are popped, the memory stays allocated until the entire allocation becomes unused.
Always clearing the capacity of the various datastructures we have would alleviate that, but most likely negatively impact performance.
@nivkner Could you try running as cargo miri run -- -Zmiri-disable-validation
to check if the memory usage is caused by Stacked Borrows?
However, Stacked Borrows should re-use memory when the items are popped and new items are pushed, so unless the stacks are for some reason growing really large I don't think this is it. My bet would be on the dead_alloc_map
, which will just keep growing when things get allocated and deallocated again. There are probably clever ways to compress this...
I wonder if there is a good way to see which buffers are the ones that are growing so big.
@RalfJung I ran the example with -Zmiri-disable-validation
and the memory usage did not grow over time like with validation enabled
Thanks! Looks like my bet was wrong then.^^
That's still somewhat odd, why would it just keep growing?
It's definitely not memory still being used after the borrow stacks got popped, adding a shrink_to_fit
does not make any noticeable difference (in the original example with serde).
https://github.com/solson/miri/pull/597 should help a lot -- on my system it cuts memory usage by 90%. Validation still almost doubles memory consumption, but that's way better than the factor of 15 or so that it used to cost before.^^
@nivkner you had another comment here that you deleted, saying my PR introduced a severe perf regression. Now the comment is gone. What's up with that?
there is still a regression, here is the code to reproduce:
static EXPECTED: &[u8] = &[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 255, 255, 0, 0, 255, 255, 0, 0, 255, 255, 0, 0, 255, 255, 0, 0, 255, 255, 0, 0, 255, 255, 0, 0, 255, 255, 0, 0, 255, 255, 0, 0, 255, 255, 0, 0, 255, 255, 0, 0, 255, 255, 0, 0, 255, 255, 0, 0, 255, 255, 0, 0, 0, 0, 0, 0, 255, 255, 0, 0, 255, 255, 0, 0, 255, 255, 0, 0, 255, 255, 0, 0, 255, 255, 0, 0, 255, 255, 0, 0, 255, 255, 1, 0, 255, 255, 1, 0, 255, 255, 1, 0, 255, 255, 1, 0, 255, 255, 0, 0, 255, 255, 0, 0, 255, 255, 0, 0, 255, 255, 0, 0, 254, 255, 0, 0, 254, 255, 0, 0, 254, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 0, 0, 255, 255, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 255, 255, 1, 0, 255, 255, 1, 0, 0, 0, 1, 0, 255, 255, 2, 0, 255, 255, 2, 0, 255, 255, 2, 0, 255, 255, 2, 0, 255, 255, 2, 0, 255, 255, 1, 0, 255, 255, 1, 0, 255, 255, 0, 0, 254, 255, 0, 0, 254, 255, 0, 0, 254, 255, 255, 255, 254, 255, 254, 255, 254, 255, 253, 255, 253, 255, 253, 255, 253, 255, 252, 255, 254, 255, 251, 255, 254, 255, 251, 255, 254, 255, 252, 255, 255, 255, 252, 255, 0, 0, 252, 255, 0, 0, 252, 255, 1, 0, 252, 255, 1, 0, 252, 255, 2, 0, 252, 255, 2, 0, 252, 255, 2, 0, 252, 255, 2, 0, 252, 255, 2, 0, 253, 255, 1, 0, 252, 255, 0, 0, 252, 255, 255, 255, 251, 255, 0, 0, 251, 255, 0, 0, 251, 255, 0, 0, 252, 255, 2, 0, 252, 255, 3, 0, 252, 255, 4, 0, 253, 255, 5, 0, 254, 255, 5, 0, 253, 255, 6, 0, 253, 255, 6, 0, 253, 255, 5, 0, 253, 255, 5, 0, 254, 255, 4, 0, 254, 255, 3, 0, 251, 255, 0, 0, 250, 255, 255, 255, 253, 255, 254, 255, 252, 255, 252, 255, 247, 255, 251, 255, 247, 255, 252, 255, 252, 255, 254, 255, 252, 255, 254, 255, 252, 255, 255, 255, 254, 255, 1, 0, 1, 0, 1, 0, 4, 0, 2, 0, 8, 0, 2, 0, 12, 0, 1, 0, 13, 0, 0, 0, 12, 0, 0, 0, 11, 0, 255, 255, 8, 0, 254, 255, 7, 0, 0, 0, 7, 0, 253, 255, 11, 0, 248, 255, 15, 0, 247, 255, 17, 0, 250, 255, 17, 0, 251, 255, 13, 0, 253, 255, 7, 0, 255, 255, 3, 0, 255, 255, 254, 255, 255, 255, 252, 255, 255, 255, 252, 255, 254, 255, 250, 255, 255, 255, 242, 255, 254, 255, 239, 255, 252, 255, 248, 255, 255, 255, 249, 255, 5, 0, 239, 255, 7, 0, 238, 255, 10, 0, 249, 255, 18, 0, 254, 255, 25, 0, 253, 255, 27, 0, 0, 0, 31, 0, 4, 0, 34, 0, 4, 0, 34, 0, 8, 0, 36, 0, 8, 0, 37, 0, 2, 0, 36, 0, 4, 0, 34, 0, 8, 0, 28, 0, 3, 0, 15, 0, 255, 255, 11, 0, 0, 0, 12, 0, 251, 255, 8, 0, 252, 255, 10, 0, 0, 0, 23, 0, 252, 255, 31, 0, 248, 255, 30, 0, 254, 255, 30, 0, 255, 255, 26, 0, 250, 255, 22, 0, 250, 255, 20, 0, 244, 255, 15, 0, 237, 255, 10, 0, 246, 255, 13, 0, 242, 255, 6, 0, 213, 255, 243, 255, 213, 255, 240, 255, 247, 255, 244, 255, 246, 255, 227, 255, 214, 255, 216, 255, 219, 255, 228, 255, 251, 255, 235, 255, 1, 0, 232, 255, 248, 255, 236, 255, 4, 0, 238, 255, 26, 0, 232, 255, 44, 0, 230, 255, 66, 0, 226, 255, 86, 0, 219, 255, 88, 0, 215, 255, 72, 0, 210, 255, 50, 0, 225, 255, 28, 0, 23, 0, 14, 0, 64, 0, 16, 0, 51, 0, 26, 0, 32, 0, 34, 0, 39, 0, 42, 0, 48, 0, 35, 0, 58, 0, 255, 255, 72, 0, 220, 255, 69, 0, 197, 255, 58, 0, 158, 255, 54, 0, 132, 255, 36, 0, 153, 255, 12, 0, 146, 255, 5, 0, 83, 255, 237, 255, 110, 255, 197, 255, 252, 255, 214, 255, 51, 0, 1, 0, 233, 255, 250, 255, 226, 255, 250, 255, 45, 0, 46, 0, 47, 0, 70, 0, 6, 0, 55, 0, 19, 0, 60, 0, 38, 0, 62, 0, 42, 0, 47, 0, 61, 0, 46, 0, 40, 0, 42, 0, 237, 255, 22, 0, 222, 255, 6, 0, 221, 255, 206, 255, 195, 255, 115, 255, 219, 255, 85, 255, 17, 0, 93, 255, 26, 0, 76, 255, 46, 0, 102, 255, 80, 0, 193, 255, 48, 0, 252, 255, 18, 0, 20, 0, 50, 0, 47, 0, 58, 0, 53, 0, 44, 0, 61, 0, 57, 0, 85, 0, 37, 0, 80, 0, 0, 0, 86, 0, 248, 255, 106, 0, 161, 255, 49, 0, 43, 255, 248, 255, 125, 255, 47, 0, 49, 0, 63, 0, 40, 0, 217, 255, 187, 255, 182, 255, 219, 255, 236, 255, 63, 0, 244, 255, 58, 0, 242, 255, 244, 255, 25, 0, 225, 255, 41, 0, 11, 0, 45, 0, 76, 0, 47, 0, 167, 0, 5, 0, 5, 1, 219, 255, 21, 1, 173, 255, 183, 0, 84, 255, 35, 0, 134, 255, 177, 255, 138, 0, 186, 255, 10, 1, 69, 0, 124, 0, 228, 0, 0, 0, 135, 1, 227, 255, 82, 2, 172, 255, 190, 2, 178, 255, 115, 2, 248, 255, 39, 2, 243, 255, 253, 1, 13, 0, 116, 1, 120, 0, 96, 1, 125, 0, 110, 2, 127, 0, 179, 2, 223, 0, 106, 1, 126, 0, 130, 1, 223, 255, 147, 3, 198, 0, 190, 3, 201, 1, 200, 1, 42, 1, 244, 1, 233, 0, 3, 4, 213, 1, 72, 4, 170, 1, 150, 3, 160, 0, 43, 4, 141, 0, 196, 4, 189, 0, 221, 4, 164, 0, 95, 5, 41, 1, 98, 5, 247, 1, 19, 5, 190, 2, 14, 6, 161, 3, 7, 7, 87, 3, 216, 6, 35, 2, 38, 7, 90, 2, 136, 7, 64, 3, 200, 6, 28, 3, 199, 6, 165, 3, 169, 7, 105, 5, 143, 7, 26, 6, 57, 8, 205, 5, 156, 10, 169, 5, 132, 11, 25, 5, 208, 10, 181, 4, 156, 10, 66, 5, 227, 9, 170, 5, 166, 9, 117, 6, 45, 12, 63, 8, 42, 13, 128, 8, 136, 10, 155, 7, 109, 11, 1, 9, 6, 15, 98, 10, 121, 9, 136, 8, 147, 252, 189, 7, 43, 247, 63, 10, 147, 249, 92, 11, 172, 248, 172, 10, 112, 245, 137, 11, 76, 246, 44, 12, 184, 247, 138, 11, 118, 246, 144, 12, 94, 246, 171, 13, 112, 247, 162, 12, 168, 246, 33, 13, 63, 246, 29, 15, 226, 247, 188, 14, 190, 248, 75, 15, 238, 247, 86, 17, 19, 247, 118, 11, 10, 247, 232, 254, 238, 247, 30, 249, 56, 248, 124, 250, 6, 247, 1, 250, 161, 246, 3, 249, 81, 247, 117, 250, 60, 247, 202, 250, 212, 247, 60, 250, 15, 249, 140, 250, 34, 248, 221, 249, 105, 247, 218, 249, 205, 248, 113, 251, 138, 248, 90, 250, 41, 248, 230, 248];
static PCM: &[i16] = &[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, -2, 0, -2, 0, -2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 2, 0, 2, 0, 2, 0, 2, 0, 2, 0, 1, 0, 1, 0, 0, -2, 0, -2, 0, -2, 0, -2, -2, -2, -3, -3, -3, -3, -4, -2, -5, -2, -5, -2, -4, 0, -4, 0, -4, 0, -4, 1, -4, 1, -4, 2, -4, 2, -4, 2, -4, 2, -4, 2, -3, 1, -4, 0, -4, 0, -5, 0, -5, 0, -5, 0, -4, 2, -4, 3, -4, 4, -3, 5, -2, 5, -3, 6, -3, 6, -3, 5, -3, 5, -2, 4, -2, 3, -5, 0, -6, 0, -3, -2, -4, -4, -9, -5, -9, -4, -4, -2, -4, -2, -4, 0, -2, 1, 1, 1, 4, 2, 8, 2, 12, 1, 13, 0, 12, 0, 11, 0, 8, -2, 7, 0, 7, -3, 11, -8, 15, -9, 17, -6, 17, -5, 13, -3, 7, 0, 3, 0, -2, 0, -4, 0, -4, -2, -6, 0, -14, -2, -17, -4, -8, 0, -7, 5, -17, 7, -18, 10, -7, 18, -2, 25, -3, 27, 0, 31, 4, 34, 4, 34, 8, 36, 8, 37, 2, 36, 4, 34, 8, 28, 3, 15, 0, 11, 0, 12, -5, 8, -4, 10, 0, 23, -4, 31, -8, 30, -2, 30, 0, 26, -6, 22, -6, 20, -12, 15, -19, 10, -10, 13, -14, 6, -43, -13, -43, -16, -9, -12, -10, -29, -42, -40, -37, -28, -5, -21, 1, -24, -8, -20, 4, -18, 26, -24, 44, -26, 66, -30, 86, -37, 88, -41, 72, -46, 50, -31, 28, 23, 14, 64, 16, 51, 26, 32, 34, 39, 42, 48, 35, 58, 0, 72, -36, 69, -59, 58, -98, 54, -124, 36, -103, 12, -110, 5, -173, -19, -146, -59, -4, -42, 51, 1, -23, -6, -30, -6, 45, 46, 47, 70, 6, 55, 19, 60, 38, 62, 42, 47, 61, 46, 40, 42, -19, 22, -34, 6, -35, -50, -61, -141, -37, -171, 17, -163, 26, -180, 46, -154, 80, -63, 48, -4, 18, 20, 50, 47, 58, 53, 44, 61, 57, 85, 37, 80, 0, 86, -8, 106, -95, 49, -213, -8, -131, 47, 49, 63, 40, -39, -69, -74, -37, -20, 63, -12, 58, -14, -12, 25, -31, 41, 11, 45, 76, 47, 167, 5, 261, -37, 277, -83, 183, -172, 35, -122, -79, 138, -70, 266, 69, 124, 228, 0, 391, -29, 594, -84, 702, -78, 627, -8, 551, -13, 509, 13, 372, 120, 352, 125, 622, 127, 691, 223, 362, 126, 386, -33, 915, 198, 958, 457, 456, 298, 500, 233, 1027, 469, 1096, 426, 918, 160, 1067, 141, 1220, 189, 1245, 164, 1375, 297, 1378, 503, 1299, 702, 1550, 929, 1799, 855, 1752, 547, 1830, 602, 1928, 832, 1736, 796, 1735, 933, 1961, 1385, 1935, 1562, 2105, 1485, 2716, 1449, 2948, 1305, 2768, 1205, 2716, 1346, 2531, 1450, 2470, 1653, 3117, 2111, 3370, 2176, 2696, 1947, 2925, 2305, 3846, 2658, 2425, 2184, -877, 1981, -2261, 2623, -1645, 2908, -1876, 2732, -2704, 2953, -2484, 3116, -2120, 2954, -2442, 3216, -2466, 3499, -2192, 3234, -2392, 3361, -2497, 3869, -2078, 3772, -1858, 3915, -2066, 4438, -2285, 2934, -2294, -280, -2066, -1762, -1992, -1412, -2298, -1535, -2399, -1789, -2223, -1419, -2244, -1334, -2092, -1476, -1777, -1396, -2014, -1571, -2199, -1574, -1843, -1167, -1910, -1446, -2007, -1818];
fn main() {
for _ in 0..10 {
mse(PCM.len(), PCM, EXPECTED);
}
}
fn read_i16(buffer: &[u8], index: usize) -> i16 {
const SIZE: usize = std::mem::size_of::<i16>();
let mut bytes: [u8; SIZE] = [0u8; SIZE];
bytes.copy_from_slice(&buffer[(index * SIZE)..(index * SIZE + SIZE)]);
unsafe { std::mem::transmute(bytes) }
}
fn mse(samples: usize, frame_buf: &[i16], buf_ref: &[u8]) -> f64 {
let mut MSE = 0.0;
let max_samples = std::cmp::min(buf_ref.len() / 2, samples as usize);
for i in 0..max_samples {
let ref_res = read_i16(buf_ref, i);
let info_res = frame_buf[i as usize];
let diff = (ref_res - info_res).abs();
MSE += f64::from(diff.pow(2));
}
MSE / max_samples as f64
}
sorry for deleting
so this code has gotten worse? Or it's just still bad?
This latest one has gotten worse with the PR.
I think what's happening is that it just tries merging things all the time when there is nothing to merge...
i don't think the PR solves the problem of memory usage growing over time.
you can see this by running the mse benchmark in loops:
iterations | memory (kbytes) | memory with -Zmiri-disable-validation (kbytes) |
---|---|---|
1 | 275988 | 129548 |
2 | 448164 | 131372 |
3 | 590356 | 134160 |
this means that even if the memory actually being used at any given time is small, trying to run the same function for different buffers (or the same buffer in this case) exhausts all available memory eventually.
Fair enough.
However, I am not sure if that is a bug. The issue with global static arrays is that each time you access them, you create a new pointer that is allowed to access this array -- but all previous pointers remain valid! It is globally shared, after all. So during program execution, you accumulate many many pointers that are allowed to access this memory.
You can see that already with this simple program:
static S: u8 = 0;
fn main() {
for i in 0..0xFFFF_FFFFusize {
let _val = &S;
}
}
Memory consumption just keeps growing while this program runs.
This is an inherent property of Stacked Borrows. We could do some kind of garbage collection, or maybe start approximating "old" pointers at some point, but no trivial implementation change will fix this.
when running this with
miri
, usingcargo-miri
, this example uses around 2.5 Gigabytes of ram as seen by the difference in system memory usage while executing the program.I think this is a memory leak because when simply pushing the same amount of elements (1795), there is barely a difference in memory usage.
this problem was originally found with a program that doesn't use
serde
, but this is the smallest example I found.