Database not fully updated/updating

[LoZio]

Running 9c256233210a (latest as of today) No matter what you set with SKIPSYNC environment variable (or if you set it at all), it always starts and syncs (taking a lot of time of course)

[immauss]

That's odd .... The most recent push .... tagged as 22.4-beta has the following hashes according to hub.docker.com:

c3ebd808700f linux/amd64

5a82047531f8 linux/arm/v7

e3f5171f8245 linux/arm64

The "latest" tag has:

2d2001b425ae linux/amd64

df1001a8cf8d linux/arm/v7

f2ff42eab444 linux/arm64

None of those match that hash ...

The "latest" tag is still on the 21.04 branch and should be working fine. The other ... well ... it's beta, but I've not seen any issues with the processing of that flag. It's having issues with upgrading from 21.4 DB, but otherwise works fine with a new DB.

Let me know.

-Scott

[LoZio]

[immauss]

Hmmm .... My Bad.

I had it in my head that the Image ID was the first characters from the Digest. But that does not appear to be corect.

So ... That's the same image I'm running in my prod, which is how I realized my error.

I've just tested, and it seems to work properly. Can you share the start commands and logs... maybe that will give me an idea of what's going on ..

Thanks, -Scott

[LoZio]

Where can I find them easily?

[LoZio]

Adding to the initial issue: I know it is syncing because the first processes starting are rsync, the web interface comes up late and when I go to feed they are always in updating state for some 10/15 minutes.

[immauss]

Are you starting with docker-compose or just on the command line?

[immauss]

if command line, the command you are using, if compose, your docker-compose.yaml will help.

-Scott

[LoZio]

Command line docker docker run --detach --publish 8087:9392 -e PASSWORD=xxxxxxx -e HTTPS=true -e GSATIMEOUT=60 -e SKIPSYNC=true --volume openvas:/data --name openvas-2022-08-21 immauss/openvas:latest Just to be clear I'm using SKIPSYNC since you added it and always worked fine untile methinks last update.

[immauss]

I have a theory ..... It's still skipping the actual download of the data from Greenbone, but for some reason, the data on disc and the what's in the DB are not in sync. This causes gvmd to go through the whole process of reading every NVT and pushing it to the database. I'm testing now, but also running a new refresh.

I'll let you know.

Thanks, Scott

[immauss]

OK ... so the timeout in my script to refresh the database was too short. So the feeds themselves were getting updated, but gvmd didn't have enough time to actually update the database .... Looks like it's taking around 45 minutes now. So ... I've upped the time from 40m to 60. It's running again now, so hopefully in an hour or so (depending on the rebuild time), it should be a good fresh latest image uploaded.

[immauss]

@LoZio Thanks. This is resolved now and hopefully will not reoccur.

latest was DB updated in <5m and ready to scan in <15m

-Scott

[LoZio]

I updated and can confirm it started quite fast. When I manually sync'ed it took a lot to finish the process (expected) but not all the parts were updated. This is strange because 2 days ago, with the previous version, I created a brand new VM + containers and after update it was all up to date.

Of course I keep the data when I create a new container. Stats from this run (I know t's not related to your containers), on a 4 core Intel/ 8 Gb RAM / SSD dedicated VM (debian 11):

• feed update (rsync): about 2 minutes (previous was 2 days ago)
• "Update in progress..." phase on feeds page: 38 minutes

Thanks @immauss

[immauss]

That's odd .... sounds like something didn't finish on the sync. You can run it manually even if you started with SKIPSYNC:

docker exec -it <container-name> /scripts/sync.sh

If that completes with no errors, and it's still off, let me know.

-Scott

[LoZio]

I did exactly that, launched the manual sync with the script. If this helps, all of the four lines went to Updating (or something like that) during the update.

[LoZio]

Did another run now, if it helps I attached the output from the sync script vasupdate.zip

[immauss]

So that looks normal. Is it still out of sync?

[LoZio]

Yes it is, sorry I was not clear from the beginning. I can sync it any time I want, always out of sync. A clean installation syncs everything instead. I mean, if using the same image I create a new docker volume bound to the data dir, it says all is up to date. Using my old data it does not seem to sync CERT and GVMDATA.

[immauss]

Very odd this .... And I don't see anything in the logs to suggest a problem.

Have you tried any of the "--optimize". or --rebuild options with gvmd ?

-Scott

[LoZio]

Today I did this:

1. updated the image, now I got

   REPOSITORY        TAG       IMAGE ID       CREATED      SIZE
   immauss/openvas   latest    0246e1badd0f   2 days ago   1.5GB

2. created a new instance with same docker command as above
3. waited 10 minutes after start to settle. Correctly it told me it was out of date but this time CERT and GVMD data were shown at the same date as other feeds. With previous image it was not.
4. Started the sync script. All went updating
5. GVMD data was pulled correctly
6. 20 minutes were spent here
7. It ended up with all up to date

I dont' know what changed with last build, but starting from pic at 3. I suppose there is some visualization mismatch between loaded feeds and the interface, or the feeds update cycle do not finish correctly some times. When last week I was out of sync with gvmd anc CERT I restarted the container several times, and always was out of sync as in the pic in my previous message. To answer your precise question: no I didn't use --rebuild/--optimize, I discovered they exist right now ;-)

[immauss]

Well, I'm glad it's working now. If you ever figure out what the true cause was, please let me know. I really haven't made any changes other than content updates to the 21.4 image in a while. . . . .