search

Digital-Defiance / nlp-metaformer

An ablation study on the transformer network for Natural Language Processing
3 stars 0 forks source link

memory leak in pyspark worker #33

Closed RuiFilipeCampos closed 8 months ago

RuiFilipeCampos commented 8 months ago

the leak is happening directly on the java heap

I'll try to resolve it, but in all likelihood I have no control over it

the solution will likely be to periodically restart the process

RuiFilipeCampos commented 8 months ago

restarting the pyspark session should do the trick

need to change from "randomSplit" though, otherwise the splits are different on each fetch