search

snipe / snipe-it

A free open source IT asset/license management system
https://snipeitapp.com
GNU Affero General Public License v3.0
11.2k stars 3.2k forks source link

Backups are getting bigger and cannot be deleted #12773

Open Joly0 opened 1 year ago

Joly0 commented 1 year ago

Debug mode

Describe the bug

I am facing a problem where my backups keep getting bigger, which is ok so far, but they get bigger without any real reason. Current backups are about 180MB in size, but if i look into the zips the actual sql file which is in there is only 200KB in size. There are no other files in the zips other than the db-dumps folder and the dump file itself. But additionally, i cant even delete the backups somehow via the webui. I can ofc delete them manually, but on the webui the button to delete is "greyed-out" and not clickable: image

I am logged in as admin and tried other users aswell, but that didnt work.

Reproduction steps

  1. Create backup
  2. Open backup zip and check contents
  3. try to delete backup ...

Expected behavior

Backups shouldnt be this large, if there is only a small sql dump in there and they should be deletable.

Screenshots

No response

Snipe-IT Version

v6.0.14 - build 9236

Operating System

Windows

Web Server

IIS

PHP Version

8.1

Operating System

No response

Browser

No response

Version

No response

Device

No response

Operating System

No response

Browser

No response

Version

No response

Error messages

No response

Additional context

No response

snipe commented 1 year ago

If you're running backups on a cron/scheduled task, they should delete themselves eventually - but if you want to manually delete, go to your .env and set ALLOW_BACKUP_DELETE=true and you should be able to delete them yourself.

Joly0 commented 1 year ago

Ah, i see. Thank you. But any idea about my other issue? Why are my backups getting larger, although they only store the dump file?

I assume, there should be the images and so on aswell in there. But why arent these in there but the zip is still that large?

Joly0 commented 1 year ago

Ok, i have spun up a test vm with linux and installed snipe-it and there backups are working without an issue. So i guess this is a permission issue, but so far i cant see any wrong permissions. I have triple-checked all the permissions according to the documentation here https://snipe-it.readme.io/docs/windowsiis and so far, everything looks fine

Snipe-IT Documentation
Windows/IIS
Setting Up an IIS Website📘NOTE:For the purposes of this walkthrough, we are assuming you're using assets.portal.local as your Snipe-IT local domain name. This should be changed for your own installation. Extract Snipe-IT to C:\inetpub\wwwroot\snipe-it (folder name can be changed but we will referen...
snipe commented 1 year ago

It doesn't really make sense to me either - the backups capture the DB dump and any uploaded files. Are you running ay API stuff on a cron or anything? If it's jamming in a bunch of tiny updates into the action log table, I could see that bloating things out.

Joly0 commented 1 year ago

Nope, nothing running in Cron or anything. This is a plain simple install on windows/iis. I can see, that there should be and "app" folder in the zip with all the uploaded files, next to the "db-dumps" folder, but there isnt, but still the zip has the size as if the files were actually there. That doesnt make any sense to me, but its how it is. So somehow the backup process tries to access those files and packs them in the zip, but somehow fails in doing so, which ends up with a 200MB zip with only a 5KB db-dump in it

uberbrady commented 1 year ago

that's super-weird I mean, the ZIP can only contain the files that are ... in it. Maybe hidden directories or dot-files or something? When you unzip it you should have, yes, the SQL file, but also every uploaded file from every section.

Joly0 commented 1 year ago

that's super-weird I mean, the ZIP can only contain the files that are ... in it. Maybe hidden directories or dot-files or something? When you unzip it you should have, yes, the SQL file, but also every uploaded file from every section.

Hey, no hidden files/directories or dot-files. This is plain windows territory here and everything is visible as it should. I just tried and installed snipe-it fresh, moved my files over and run the php artisan snipeit:backup command to see what happens image I get the right output and the zip itself looks right too image but image There is just nothing other than the db-dumps

Joly0 commented 1 year ago

Any ideas on this one @uberbrady (sry for the ping) ? I have tried everything i can. I checked folder permissions like 10x, i have event tried reinstalling snipe-it and still same issue. I have no idea anymore so far. Not sure when this issue started, might be the moment i switched from v5 to v6 and updated to php 8.1 from 7.4, but i cant tell for sure

Maybe any php config issues, i can look out for? Any php commands i can run to check for correct settings or something?

marcusmoore commented 1 year ago

We use Spatie's backup package and I found an issue and discussion in that repository from someone that is having the same problem.

@Joly0 can you try to add relative_path to your backup config file and set it to storage_path() and see if you have better results?

'relative_path' => storage_path()

Joly0 commented 1 year ago

We use Spatie's backup package and I found an issue and discussion in that repository from someone that is having the same problem.

@Joly0 can you try to add relative_path to your backup config file and set it to storage_path() and see if you have better results?

'relative_path' => storage_path()

Hey thank your for sharing your information. Could you please explain a bit clearer to me, where to change what? I am not quite understanding, what exactly you mean

marcusmoore commented 1 year ago

@Joly0 no problem. The package that we use added a configuration option that we don't have in our configuration file.

In the configuration file there is an option in the backup.source.files array called relative_path. We don't have that option exposed in our application but our app will pick it up and use it if added.

So I was hoping you could add 'relative_path' => storage_path() directly under ignore_unreadable_directories in the configuration file located at config/backup.php then run a backup and let us know if it fixes the issue. If it does then we can include it in our application in a future release.

To double-check: you ran through the Fix Permissions section of the Windows docs right?

Joly0 commented 1 year ago

Ok, thank your for the explanation. I added the the code to the file as shown here image but unfortunately it did not solve my issue.

I checked the windows permissions like 10 times now. It was already working for about 3 years without an issue, but i assume after upgrading to php 8 and snipe-it 6 something broke

marcusmoore commented 1 year ago

@Joly0 I missed it but it looks like the backup package does not completely support Windows. (Someone else mentions the same issue that you are having)

We'll update the documentation and see if we can submit a patch to the upstream package.

Joly0 commented 1 year ago

Hm, but i am curious why this started happening now? Did the package thats used for backups changed recently? As i said, it did work for the past 3 years now and stopped recently (probably when switching to snipe-it 6 and php 8)

marcusmoore commented 1 year ago

I'm honest not sure and it doesn't look like the package maintainers are either.

The backup package has remained the same but if you went from v5.4.4 to v6.0.14 then the backup package was updated from v6.11.1 to v6.16.5. I did a very quick scan of the diff between those versions the use of DIRECTORY_SEPARATOR caught my eye, particularly in the FileSelection and Zip classes.

I'd be curious if downgrading to v6.11.1 fixes the issue but it would require dropping back down to PHP 7 since that version doesn't include PHP 8 support 😕

Joly0 commented 1 year ago

I am curious, is there any idea how to fix this then? Will the package thats used for backup be replaced or will the function be somehow modified to support windows or what is the plan?

I dont know how large the windows user base of snipe-it is and and how "critical" this issue is for others, but i guess there might be others, who use backups and dont know, that their backups are missing files. So i'd assume this isnt that big of an issue, but i thought i could still ask, if there are any plans to fix it :)

uberbrady commented 1 year ago

I don't know if you'd be willing, but I'd love to take an actual look at the dumpfile - there has to be something in there that's taking up all that space. My guesses were that somehow your backups are including all of your previous backups in the backup? Or that maybe some hidden temporary files that are being generated are not getting correctly cleaned up. But there's gotta be something there.

And I totally understand if you're not able to share that backup file with me, I'd get that. If not, If there's any way you can look at a lower-level into the zipfile and figure out where all the space is getting used up, that would help us out a lot. Then we can maybe figure out what's going on and why.

Like, if you unzip the zip, does the resulting folder take up a huge amount of space? Or is it tiny? If the resulting folder is tiny, then I feel like we have some other kind of weird problem upon which I couldn't even speculate...

Joly0 commented 1 year ago

I mean, sure i guess i could share a backup file with you if you'd tell me how i can contact you, i could probably send you the file

uberbrady commented 1 year ago

That'd be great! Ping me on Discord and ping me or @snipe and we'll give you an email adress to send it to. We'll poke through the file, but won't look at the contents, and will promptly delete it.

uberbrady commented 1 year ago

Okay, just to keep this issue updated. @Joly0 was able to get me a copy of his backup file, and there is stuff in there. The reason it keeps getting bigger is because various EULA's keep getting signed, and signature PNG's getting created, over time. And the backup system tries to grab everything - including those things.

But it puts them in very silly-named directories. When I extract the Zip on my Mac, I get 4 'directories' - with these sizes:

8.0K    X:
8.9M    X:\path\to\inetpub\wwwroot\snipe-it\public
201M    X:\path\to\inetpub\wwwroot\snipe-it\storage
2.1M    db-dumps

(I redacted some path information just to be on the safe side)

Windows Explorer's zip-viewer can't see those drive-letter-prefixed directories at all, but @Joly0 was able to open the ZIP archive directly in 7zip, and then he was able to see them.

So it seems like something has definitely changed on the laravel-backup side, and something is going to need to get fixed there as well. I'm going to update that issue too and see if we can get a little bit of forward progress on this.

marcusmoore commented 1 year ago

Thanks for the update Brady 😄

For reference, here is the discussion to keep an eye on: https://github.com/spatie/laravel-backup/discussions/1679

GitHub
Backup failed because The dump process failed with exitcode 2 · spatie/laravel-backup · Discussion #1679
Hey, i am using snipe-it which uses laravel-backup and i have an issue open over there aswell (snipe/snipe-it#12852) but it doesnt seem like i will get any help. I did some testing and the mysqldum...