search

Azure / WALinuxAgent

Microsoft Azure Linux Guest Agent
http://azure.microsoft.com/
Apache License 2.0
541 stars 372 forks source link

waagent downloads updates to /var/lib/waagent but doesn't remove the old .zip files #2489

Open carlosporter opened 2 years ago

carlosporter commented 2 years ago

Description: Any Linux OS running waagent for a few months will have at least a few .zip files at /var/lib/waagent that corresponds to older versions of the WALinuxAgent file that is downloaded every time Azure pushes a new update.

Over time these files start to pile up and add up to unnecessary space utilization on the root file syste or /var if LVM is in use.

The idea is to include a log rotation config file to automatically find and delete these files or even adding a cron job to this would be fine.

It's easy to check this on any server that has been running waagent for more that a few months ( about 6-8 months ).

Just run:

/usr/bin/find /var/lib/waagent -name "WALinuxAgent*.zip" -type f -exec ls -ltr {} \+

An you will see all the old updates downloaded by the waagent

As an example, an easy way to fix this is by adding a log rotate config file with these commands

cat << EOF > /etc/logrotate.d/waagent.cleanup
/var/lib/waagent/*.zip {
        # Ignore if there are no zip files at /var/lib/waagent/
        missingok

        # Don't send email
        nomail

postrotate
        # Remove any file that matches WALinuxAgent*.zip and its older than 60 days
        /usr/bin/find /var/lib/waagent -name "WALinuxAgent*.zip" -type f -mtime +60 -exec rm {} \;
endscript
}
EOF
johanburati commented 2 years ago

How many zip files do you see ?

FYI Marco already reported the issue in the past for WALinuxAgent https://github.com/Azure/WALinuxAgent/issues/2031 and I did for the extensions https://github.com/Azure/WALinuxAgent/issues/1730

larohra commented 2 years ago

Hey @carlosporter, thanks for reaching out to us.

The idea is to include a log rotation config file to automatically find and delete these files or even adding a cron job to this would be fine.

We already have a similar logic like this in the Linux Guest Agent service where we periodically (~1hr) check for the zips and delete any of the ones that were deleted from the upstream server. This should delete all the older versions that are no longer in use. In addition to this, a new feature is being added in the Guest Agent which will only download just the current version in /var/lib/waagent and delete everything else, that will fix all these issues once and for all.

That being said, you still shouldn't see any zip files < v2.3.1.1. If you're seeing that, then could you please share the contents of /var/lib/waagent/ directory and if possible also share the /var/log/waagent.log file please.

carlosporter commented 2 years ago

@johanburati

How many zip files do you see ? I already deleted the ones I saw on my old server, but on the one I deployed today I can see some old update files already.

[root@vm3372gesmd3302 waagent]# ll
total 137824
-rw-r--r--. 1 root root       875 Feb  2 19:41 0B18A7E9A779B00446A174A1AA17C948D2E02A37.crt
-r--------. 1 root root      4904 Feb  2 19:41 Certificates.p7m
-r--------. 1 root root      4015 Feb  2 19:41 Certificates.pem
-rw-r--r--. 1 root root      5075 Feb  2 19:41 Certificates.xml
-rw-r--r--. 1 root root      1280 Feb  2 19:41 EB0DA963942DC59C1622235FB430CCF911BA908B.crt
-rw-r--r--. 1 root root      1860 Feb  2 19:41 EB0DA963942DC59C1622235FB430CCF911BA908B.prv
-rw-r--r--. 1 root root      3506 Feb  2 19:41 ExtensionsConfig.2.xml
-rw-r--r--. 1 root root      2134 Feb  2 19:41 GoalState.2.xml
-rw-r--r--. 1 root root      1736 Feb  2 19:41 HostingEnvironmentConfig.xml
-rw-r--r--. 1 root root         1 Feb  2 19:41 Incarnation
drwxr-xr-x. 8 root root      4096 Feb  2 19:42 Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux-1.14.9
-rw-r--r--. 1 root root     22723 Feb  2 19:41 Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux.2.manifest.xml
-rw-r--r--. 1 root root 138322516 Feb  2 19:41 Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux__1.14.9.zip
-rw-r--r--. 1 root root      5303 Feb  3 03:32 Prod.2.manifest.xml
-rw-r--r--. 1 root root        12 Feb  2 19:32 Protocol
-rw-r--r--. 1 root root       919 Feb  2 19:41 SharedConfig.xml
-r--------. 1 root root      1107 Feb  2 19:32 TransportCert.pem
-r--------. 1 root root      1704 Feb  2 19:32 TransportPrivate.pem
-rw-r--r--. 1 root root      2888 Feb  2 19:41 VmSettings.12486027708972659514.json
drwxr-xr-x. 3 root root        65 Feb  2 19:32 WALinuxAgent-2.3.1.1
-rw-r--r--. 1 root root    643890 Feb  2 19:32 WALinuxAgent-2.3.1.1.zip
drwxr-xr-x. 3 root root        65 Feb  2 19:32 WALinuxAgent-2.4.0.2
-rw-r--r--. 1 root root    658281 Feb  2 19:32 WALinuxAgent-2.4.0.2.zip
drwxr-xr-x. 3 root root        65 Feb  2 19:32 WALinuxAgent-2.5.0.2
-rw-r--r--. 1 root root    668498 Feb  2 19:32 WALinuxAgent-2.5.0.2.zip
drwxr-xr-x. 3 root root        65 Feb  2 19:32 WALinuxAgent-2.6.0.2
-rw-r--r--. 1 root root    681680 Feb  2 19:32 WALinuxAgent-2.6.0.2.zip
-rw-r--r--. 1 root root        13 Feb  2 19:32 WireServerEndpoint
-rw-r--r--. 1 root root        40 Feb  2 19:32 event_status.json
drwx------. 2 root root         6 Feb  3 04:23 events
drwxr-xr-x. 2 root root       110 Feb  2 20:02 history
-rw-r--r--. 1 root root        72 Feb  2 19:42 initial_goal_state
-r--------. 1 root root      1519 Feb  2 19:31 ovf-env.xml
-rw-r--r--. 1 root root         2 Feb  2 19:32 partition
-rw-r--r--. 1 root root        36 Feb  2 19:32 provisioned
-rw-r--r--. 1 root root        15 Feb  2 19:32 published_hostname
-rw-r--r--. 1 root root       553 Feb  2 19:32 waagent-network-setup.py
-rw-r--r--. 1 root root      1410 Feb  3 04:23 waagent_status.2.json
[root@vm3372gesmd3302 waagent]# waagent --version
WALinuxAgent-2.2.45 running on centos 7.8.2003
Python: 2.7.5
Goal state agent: 2.6.0.2
[root@vm3372gesmd3302 waagent]# cat /etc/os-release
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

[root@vm3372gesmd3302 waagent]# rpm -qa | grep -i walinux
WALinuxAgent-2.2.45-1.el7.noarch
[root@vm3372gesmd3302 waagent]#

Also, please note that while the WALinuxAgent updates are very small, a single extension .zip file weights 132M, so some tidy up seems to be required.

[root@vm3372gesmd3302 waagent]# du -hcs *zip
132M    Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux__1.14.9.zip
632K    WALinuxAgent-2.3.1.1.zip
644K    WALinuxAgent-2.4.0.2.zip
656K    WALinuxAgent-2.5.0.2.zip
668K    WALinuxAgent-2.6.0.2.zip
135M    total
[root@vm3372gesmd3302 waagent]#

@larohra

That being said, you still shouldn't see any zip files < v2.3.1.1.

I'm sorry, do you mean that I shouldn't see any update file for any version before v2.3.1.1 ? or did you mean any version greater than v2.3.1.1 ?

If you're seeing that, then could you please share the contents of /var/lib/waagent/ directory and if possible also share the /var/log/waagent.log file please.

Yes, I can see the file on /var/lib/waagent ( it's my answer to @johanburati )

I have also uploaded the log file for the waagent.

Let me know if you need anything else.

vm3372gesmd3302_waagent.log.zip

larohra commented 2 years ago

@carlosporter

I'm sorry, do you mean that I shouldn't see any update file for any version before v2.3.1.1 ? or did you mean any version greater than v2.3.1.1 ?

I meant you shouldn't see any WALinuxAgent-2.*.zip files less than v2.3.1.1 (which I see is the case as you pointed it out in the contents for /var/lib/waagent).

Yes, I can see the file on /var/lib/waagent ( it's my answer to @johanburati )

This is by design. As mentioned above, we've made a change recently where we'll only keep the zip of the version that's currently executing, others will be deleted automatically. Since that fix has some external dependencies, I'll also send a PR clean up all but the last 2 versions of the GA. That should be relatively faster release.

Also, please note that while the WALinuxAgent updates are very small, a single extension .zip file weights 132M, so some tidy up seems to be required.

That is how large the OMS extension is unfortunately. if this is concerning please create a bug on their repo to track it separately.

I'll keep this issue open and use it for tracking the fix mentioned above. Its not technically a bug as it was designed this way so I'll treat is as an enhancement. Hope that helps!

carlosporter commented 2 years ago

Thanks @larohra

I appreciate your help with this

:-)