Closed max810 closed 1 year ago
Hi,
You can remove add_memory
: https://github.com/hkchengrex/XMem/blob/139a7458b84a4829cb3bc7434cea1e7383eca5f8/inference/inference_core.py#L99-L100
This would disable updates to the working/long-term memory while keeping the sensory memory updated.
You can also remove this entire block: https://github.com/hkchengrex/XMem/blob/139a7458b84a4829cb3bc7434cea1e7383eca5f8/inference/inference_core.py#L96-L105
This would additionally disable deep updates to the sensory (and it will run faster). Regular update to the sensory memory is still performed. Feel free to let me know if doing these give you errors.
(2) - yeah makes sense to me.
Note that the working memory will then be frozen at the end of the video. When the video is re-run from the beginning, it is no longer a "high-resolution buffer from the recent past".
Thanks, I'll try that and let you know if it works.
And you made a good point about the working memory. Regarding that, I have a question:
In the paper you state that, quote, "The first frame (with user-provided ground-truth) and the most recent $T{min} − 1$ memory frames will be kept in the working memory as a high-resolution buffer while the remainder ( $T{max} − T_{min}$ frames) are candidates for being converted into long-term memory representations" (part 3.3, second paragraph on arxiv)
If I provide multiple annotations for the video (e.g. frame 0, frame 10 and frame 30), I would want to keep all of them in the working memory forever, right? Because these are 100% correct masks and contain valuable info for the decoder.
Does the code automatically allow for that (keeping feature maps with provided annotations in the working memory forever besides frame 0)? Or I should implement it myself?
I assume the part of the code responsible for selecting candidates is inference/memory_manager.py#L234, and by specifiying HW
as a start
value, we skip the first feature map stored in the working memory? So, if I want to store multiple feature maps there, I would slice the working memory multiple times using the same KeyValueMemoryStore.get_all_sliced
method?
Also, do I understand correctly that the order of feature maps in both working and long-term memory does not matter? E.g. if we shuffle them, the predictions would not change at all?
HW
term to always keep the first frame in memory. There should be multiple ways in which you can keep more than one frame. The one that you mentioned sounds reasonable.Hi @max810, I'm having similar issue with providing and keeping multiple annotations in working memory. Did you manage to solve it?
Hi @jelenaam,
No, sorry, we're still working on it, but so far it requires quite a lot of code modifications, so it will definitely takes some time to finish.
Hi @jelenaam,
So yeah, we did solve this problem now, and made a paper about it: https://github.com/max810/XMem2
XMem++ is a wrapper around XMem (so using the same model), but it supports using multiple ground truth annotations (and keeping them in the memory permanently) out of the box now.
If you also want to disable updates to temporary memory at all (i.e. just not use it), then please look at inference_core.py#L135, just pass False
for all frames.
Big thanks to @hkchengrex for his explanations!
Hi @max810, Congrats on your paper! Thank you for letting me know about it.
Hi!
I'm been doing some experiments and have the following use-case:
I have 2 questions:
I would appreciate your advice here.
Thanks in advance!