numworks / epsilon

Modern graphing calculator operating system.
https://www.numworks.com/resources/engineering/software/
1.75k stars 466 forks source link

[Python] Add GC library to python #2202

Open yannis300307 opened 5 months ago

yannis300307 commented 5 months ago

Problem you'd like to fix

Sometimes, we need to allocate a lot of memory, but if the garbage collector doesn't delete old variables quickly enough, we can run into a memory allocation error.

Describe the solution you'd like

Add the gc lib to python to free the memory.

Describe alternatives you've considered

Make the del keyword forcing the garbage collection.

adri1numworks commented 5 months ago

Thanks for your suggestion. The gc lib is a bit too advanced for our students needs, we don't plan on adding it in future versions at the moment.

Make the del keyword forcing the garbage collection.

Do you have examples where del function is not working as you want it to ?

yannis300307 commented 5 months ago

I've made a project in which I have to store images as bytes. Sometimes, the memory is not cleared quickly enough between the images' loading despite the use of del. Moreover, the memory is not always cleared between programs executions. You can see this project here. It makes the emulator bugging, and it crashes on actual hardware (N0110) after 1 or 2 lignes rendered.

EmilieNumworks commented 5 months ago

MicroPython's garbage collector is triggered when the allocator runs out of memory. There shouldn't be such things as "it doesn't delete old variables quickly enough". The objects in the pool aren't deleted if some pointers still refer to them... What you're describing might be a bug but I cannot reproduce it (neither on the emulator nor the actual devices). Could you give the versions of your device or a video of the emulator bugging?

yannis300307 commented 5 months ago

I've a N0110 and the OS version is 22.2.0. I will send a video at the evening.

Volcanio18 commented 5 months ago

Hi, I've also encountered the same problem. You can easily reproduce this bug using this program :

a="" while True; ‎ ‎ ‎ ‎ a+="0" ‎ ‎ ‎ ‎ if len(a)%100==0: print(len(a))

You will see that every time you run this script it will run out of memory with "a" having reached a different length each time, because the ram was not cleared proprely.

yannis300307 commented 5 months ago

In this case, you never clear a during the execution, so you will always run into a MemoryAllocationError.

yannis300307 commented 5 months ago

I've reproduced the bug in this video: https://github.com/numworks/epsilon/assets/52321382/cb96bfb5-b80f-41d6-8ac4-cac5d3890c8b

As you can see, I clear a at the end of my for loop with del but the memory is not correctly cleared and it causes the error.

EmilieNumworks commented 5 months ago

Hi, I think @Volcanio18 is pointing at a heap leakage which I can indeed observe by enabling the MICROPY_MEM_STATS macro. @yannis300307 you implying that the del keyword doesn't work as intended. I'm opening issues internally for both topic and will keep you posted.

Idaill commented 3 months ago

[ FR ] Excusez moi Emilie, dans python, il est connu que "del" ne supprime rien du tout, enfin presque, il supprime l'objet ( variable ou liste etc... ) mais ne le supprime pas vraiment, il est toujours présent dans le cache jusqu'a ce que python est enfin décidé de le supprimer, le problème c'est que dans ce type de boucle, python est tellement préocupé dans la boucle while présente dans le script de Volcanio18 qu'il fini par crash car ici, même si del n'est pas utiliser, python fait en vérité en assimilant un nouveau nombre à une variable del a , a= nouvelle chose à mettre dans l'objet nommé a . Il est donc nécéssaire de rajouter le module gc pour éviter de genre d'erreur, à moins de modifier entièrement le fonctionnement de python .

Bonne chance ! Et content si j'ai pu vous aider

[ EN ] Excuse me Emilie, in python, many of us know that "del" dont delete, the only thing that this does is remove it from the existing objects list , that's all . So we need the gc module to change this, or maybe you want to change the entiere python just for this but in all cases, this isn't a bug, this is just how's is made python and micro-python .

Good luck ! And i hope i could help !

Idaill commented 3 months ago

( Oh i forgot to say for english users that if you do a=1 and after a=2 , in the cache , a=1 and a=2 are both still here even if they are not in the existing object list )

adri1numworks commented 3 months ago

Hello all, An issue was created internally and we will tackle this issue as soon as we can. Please, note that, due to prioritization reasons, it might take some time! We will keep you updated Thanks for your help