Cleanup of database not working?

[jojodat]

Hi,

I've been using your script for quite a while and it's been fantastic. However, I stumbled upon an issue after exporting the database and transforming the JSON output into an HTML page for a friend. (with the help of ChatGPT as i am not a coder)

The database cleanup process does not seem to be working correctly. Despite not being a programmer, I spotted that there is intended cleanup code. Regrettably, it either doesn't trigger or fails to execute. The database keeps growing and this has been the case through several script updates.

I've tested on various browsers and operating systems but the issue remains.

Could there be a manual trigger for the cleanup? I'm willing to help debug this. Maybe there's a way to manually activate the cleanup and provide you with the logs?

Looking forward to figuring this out together.

[MarkusSR1984]

Please check the ts_lastSeen and notSeenCounter property. ts_lastSeen is a unix timestamp notSeenCounter counts how many the Product has a timesamt older than a week and get incremented with each db cleanup. if ths vale is higer than 5(DEFAULT) the product gets removed from db.

please check this and tell me the outcome

[jojodat]

It seems it doesn't work for me. I have Products - even from potluck - which should have been removed even earlier. Like this one.

    {
        "id": REMOVED,
        "link": "/dp/B0BZTM51CF",
        "description_full": "Spigen Liquid Crystal Hülle Kompatibel mit Samsung Galaxy S23 FE -Crystal Clear",
        "description_short": "Spigen Liquid Crystal Hülle Kompatibel mit Samsung Galaxy S23 FE -Crystal Clear…",
        "data_recommendation_id": REMOVED,
        "data_recommendation_type": "VINE_FOR_ALL",
        "data_img_url": "https://m.media-amazon.com/images/I/41jSDlX1ksL._SS210_.jpg",
        "data_img_alt": "",
        "data_asin": "B0BZTM51CF",
        "data_asin_is_parent": false,
        "isFav": false,
        "isNew": false,
        "gotRemoved": false,
        "ts_firstSeen": 1705670363,
        "ts_lastSeen": 1705670363,
        "notSeenCounter": 1,
        "order_success": false,
        "generated_short": false
    },

And this is only one of the oldedst cause i did a restet of about a week ago as as it got really huge over time.

Edit: notSeenCounter is - i guess - everywhere 1. And ts_firstSeen seems to be same as ts_lastSeen everywhere.

[MarkusSR1984]

the dataset looks ok, only the id property is wrong but this got not setted to this value from AVE. The product was last seen 1 week ago and has a notSeenCounter value of 1. when the next cleanup is done it should be increased to 2. please check this.

And please remember that the db update and cleanup only works while u are an the vine website and a full cycle needs about 15 to 30 minutes.

the db cleanup is running after each completed scan.

[jojodat]

I have the website open about 16 hours or more a day. Never seen the counter with 2 or more. 🤷

btw. I also tried manual scans and to even shown the cleanup screen at the end But a cleanup never happened for me

[MarkusSR1984]

i think i found the problem. please update to Version 0.10.3.6 and recheck. and thanks for the hint of missing increase of the notSeencounter

[jojodat]

Looks good for now - already seeing many 2 and even 3 for the counter 👍

[MarkusSR1984]

please observe if the products get removed now when notSeenCounter get higher then 5 and tell me the result

[jojodat]

It worked 👍 Just cleaned up over 9k entries. Thank you very much ❤️

[jojodat]

Sorry for reopening - maybe i should have opened a new issue?

Anyway. it seems, that now the notSeenCounter is raised on every run for every product. Even for those which should still be seen. Is this intended that way?

Cause there were many products readded that were removed on the previous run

[jojodat]

Ok, seems to be fixed with latest version