search

transferwise / pipelinewise-target-redshift

Singer.io Target for Amazon Redshift - PipelineWise compatible
https://transferwise.github.io/pipelinewise/
Other
12 stars 65 forks source link

target-redshift process prevents deleted files to be removed from disk #193

Open halilduygulu opened 2 years ago

halilduygulu commented 2 years ago

Copy pasting linked issue here as they mentioned this is an plugin issue not meltano https://github.com/meltano/meltano/issues/6384

I also get same problem while using tap-zendesk + target-redshift, so it is related to this process.

Hi, when using target-redshift (wise) with meltano, I noticed disk usage on k8s node is increasing all the time even though /tmp folder and other places including / (root) folder is not growing in size.

After checking with lsof tool, you can see deleted files are kept open(or something else) by target-redshift process so they are not actually removed from disk, so I ran out of 100gb disk space on k8s node even-though pod is using less than 10gb.

Sample from output of lsof, there are 100+ files like this

target-re  160             root   54u      REG               0,79 244794493  147626721 /tmp/public-mytable_1qhpq7op.csv.3 (deleted)
target-re  160             root   55u      REG               0,79 244865304  147626722 /tmp/public-mytable_k0fcdklp.csv.4 (deleted)
target-re  160             root   56u      REG               0,79 245068385  147626736 /tmp/public-mytable_scj5v0oc.csv.2 (deleted)
target-re  160             root   57u      REG               0,79 244919191  147626725 /tmp/public-mytable_l3ozw88o.csv.3 (deleted)
target-re  160             root   58u      REG               0,79 244815828  147626726 /tmp/public-mytable_v_uerrdh.csv.4 (deleted)