[BUG]: Reveal's filling up the disk

dwilches commented 4 months ago

SDK Version

1.6.1

Client Framework

Angular

Server Platform

Java

Operating System

Linux

Description

Hello,

Today we got some errors in Reveal dashboards mentioning No space left on device:

Looking into what was filling the disk, it was temporary files created by Reveal in /tmp with names such as:

/tmp/91a58398-0ef3-4d7a-bf57-f62672be76aa2269914492392751970.tmp
/tmp/62724be1-262a-45d3-9c91-09042aee4d16256189605328959594.tmp
/tmp/615e5a2e-fa21-4456-b4e3-c65b03e3750f16547519972392649929.tmp

Some of these files were tiny, some others were several MB.

We have a disk in this machine of 27GB, of which 18GB are available for Reveal to use for the cache. This disk size was recommended previously by the Reveal team to avoid running out of space.

Could you update Reveal so it clears it's temporary files after it's done using them? or if this is part of the cache mechanism, to remove cache entries that are too old? As it is now, it's just a matter of time before a disk of any size fills up.

Thanks.

TorreyBetts commented 4 months ago

This issue has been written up and internally tracked as AB#29777. Thanks!

jorencarlson commented 4 months ago

Same issue observed in this context:

SDK Version 1.6.2-dotmatics.4

Client Framework React

Server Platform Node.js

Operating System MacOS

MLopezIG commented 4 months ago

Same issue observed in this context:

SDK Version 1.6.2-dotmatics.4

Client Framework React

Server Platform Node.js

Operating System MacOS

@jorencarlson Can you confirm what's the folder that is going out of control?

jorencarlson commented 4 months ago

The cache folder. In my case it's a custom path.

MLopezIG commented 4 months ago

The cache folder. In my case it's a custom path.

@jorencarlson If it is the cache folder, most likely is a separate issue. I suggest creating another issue here in Github to discuss it, or start a new thread in Discord.

dwilches commented 4 months ago

@TorreyBetts is there a workaround we can do while the bug is fixed? we are thinking either:

clearing the /tmp files on a schedule, but we don't know if that's safe for Reveal, as it may misbehave if we delete it's cache files.
restarting the machine on a schedule (as /tmp gets cleared on each restart, this is the one we're doing but manually, but it also affects the non-Reveal parts of the service).
having the VM replaced each time it's disk has too little empty space.

If there is something like invoking a method in Reveal we can do every day or so to clear that cache, that'd be a better workaround for now. Or if there is a way to know which files we can delete from /tmp without causing trouble to Reveal while it's running, that would also work.

Thanks

TorreyBetts commented 3 months ago

@TorreyBetts is there a workaround we can do while the bug is fixed? we are thinking either:

clearing the /tmp files on a schedule, but we don't know if that's safe for Reveal, as it may misbehave if we delete it's cache files.

restarting the machine on a schedule (as /tmp gets cleared on each restart, this is the one we're doing but manually, but it also affects the non-Reveal parts of the service).

having the VM replaced each time it's disk has too little empty space.

If there is something like invoking a method in Reveal we can do every day or so to clear that cache, that'd be a better workaround for now. Or if there is a way to know which files we can delete from /tmp without causing trouble to Reveal while it's running, that would also work.

Thanks

After consulting with the team, we suggest to use a schedule for deleting files under /tmp. This idea would be okay for Reveal as long as:

Files that have been created within the last few minutes are kept. How many minutes would be determined by how long it takes to process the data on your disk. For cleaning purposes, anything older than 1 hour old should be fine.
Do not touch the RevealCache folder. By default, it is in a folder under /tmp.

TorreyBetts commented 3 months ago

The issue with temporary files filling up the disk has been fixed and will be available in our April release.

@dwilches @jorencarlson Once you update to the latest version after the release you will need to clear out the OS temp location as this fix won't be able to remove the previously downloaded tmp files. It should be noted that most servers such as ASP.NET Core and TomCat do product tmp files, so these shouldn't be confused with the ones that Reveal had created.

We do think we covered all the possibilities here, but if you could let us know what kind of data providers you've used to do final checks. In addition to data providers, do you use client side exporting, server side exporting, data blending, etc?

dwilches commented 3 months ago

@TorreyBetts About our data providers, they are all custom JSON endpoints. And we use client side exporting and server side exporting. We do not use data blending. Thank you!

brianlagunas commented 3 months ago

Unfortunately, during testing we found an issue which is preventing us from releasing a fix with this week's release. For now, please continue to use the work-around described by Torrey. We will continue to work on this issue and provide a proper fix.

RevealBi / Reveal.Sdk