search

jef / streetmerchant

🤖 The world's easiest, most powerful stock checker
https://jef.buzz/streetmerchant
MIT License
4.97k stars 1.3k forks source link

Mediamarkt Store produces Error Status 403 #1559

Closed Lars4k closed 3 years ago

Lars4k commented 3 years ago

Expected Behavior

mediamarkt store should check the products without creating an error.

Current Behavior

Mediamarkt Store produces Error Status 403 The saturn and mediamarkt shops opertae almost identically. The saturn checker works perfectly. Mediamarkt produces 403 errors.

The Mediamarkt links open without any problems and point to the right product. Screen

Steps to Reproduce

Install latest version of streetmerchant. Tested on Windows 10 and Raspberry Pi

Environment

OS: Windows 10 / Raspberry Pi

mediamarkt.ts :

import {Store} from './store';

export const Mediamarkt: Store = {
    backoffStatusCodes: [403, 429, 503],
    currency: '€',
    labels: {
        captcha: {
            container: 'p',
            text: ['Das ging uns leider zu schnell.']
        },
        maxPrice: {
            container: 'span[font-family="price"]',
            euroFormat: false
        },
        outOfStock: [
            {
                container: '#root',
                text: ['Dieser artikel ist aktuell nicht verfügbar.']
            },
            {
                container: '#root',
                text: ['Leider keine Lieferung möglich']
            },
            {
                container: '#root',
                text: ['Nicht verfügbar']
            },
            {
                container: '#root',
                text: ['Dieser Artikel ist dauerhaft ausverkauft']
            }
        ]
    },
    links: [
        {
            brand: 'test:brand',
            model: 'test:model',
            series: 'test:series',
            url: 'https://www.mediamarkt.de/de/product/-2641856.html'
        },
        {
            brand: 'asus',
            model: 'dual',
            series: '3060ti',
            url: 'https://www.mediamarkt.de/de/product/-2701239.html'
        },
        {
            brand: 'zotac',
            model: 'twin edge',
            series: '3060ti',
            url: 'https://www.mediamarkt.de/de/product/-2704436.html'
        },
        {
            brand: 'asus',
            model: 'strix',
            series: '3070',
            url: 'https://www.mediamarkt.de/de/product/-2691243.html'
        },
        {
            brand: 'asus',
            model: 'strix oc',
            series: '3070',
            url: 'https://www.mediamarkt.de/de/product/-2691244.html'
        },
        {
            brand: 'asus',
            model: 'dual',
            series: '3070',
            url: 'https://www.mediamarkt.de/de/product/-2691245.html'
        },
        {
            brand: 'asus',
            model: 'dual oc',
            series: '3070',
            url: 'https://www.mediamarkt.de/de/product/-2691246.html'
        },
        {
            brand: 'asus',
            model: 'tuf oc',
            series: '3070',
            url: 'https://www.mediamarkt.de/de/product/-2691247.html'
        },
        {
            brand: 'gigabyte',
            model: 'gaming oc',
            series: '3070',
            url: 'https://www.mediamarkt.de/de/product/-2691439.html'
        },
        {
            brand: 'gigabyte',
            model: 'eagle oc',
            series: '3070',
            url: 'https://www.mediamarkt.de/de/product/-2691444.html'
        },
        {
            brand: 'gigabyte',
            model: 'aorus master',
            series: '3070',
            url: 'https://www.mediamarkt.de/de/product/-2696163.html'
        },
        {
            brand: 'gigabyte',
            model: 'vision oc',
            series: '3070',
            url: 'https://www.mediamarkt.de/de/product/-2696164.html'
        },
        {
            brand: 'msi',
            model: 'gaming x trio',
            series: '3070',
            url: 'https://www.mediamarkt.de/de/product/-2694894.html'
        },
        {
            brand: 'msi',
            model: 'ventus 3x oc',
            series: '3070',
            url: 'https://www.mediamarkt.de/de/product/-2694896.html'
        },
        {
            brand: 'msi',
            model: 'ventus 2x oc',
            series: '3070',
            url: 'https://www.mediamarkt.de/de/product/-2694898.html'
        },
        {
            brand: 'inno3d',
            model: 'ichill x3',
            series: '3070',
            url: 'https://www.mediamarkt.de/de/product/-2695942.html'
        },
        {
            brand: 'inno3d',
            model: 'ichill x4',
            series: '3070',
            url: 'https://www.mediamarkt.de/de/product/-2695941.html'
        },
        {
            brand: 'zotac',
            model: 'twin edge',
            series: '3070',
            url: 'https://www.mediamarkt.de/de/product/-2691365.html'
        },
        {
            brand: 'zotac',
            model: 'twin edge oc',
            series: '3070',
            url: 'https://www.mediamarkt.de/de/product/-2695671.html'
        },
        {
            brand: 'asus',
            model: 'strix',
            series: '3080',
            url: 'https://www.mediamarkt.de/de/product/-2681869.html'
        },
        {
            brand: 'asus',
            model: 'strix oc',
            series: '3080',
            url: 'https://www.mediamarkt.de/de/product/-2681871.html'
        },
        {
            brand: 'asus',
            model: 'tuf',
            series: '3080',
            url: 'https://www.mediamarkt.de/de/product/-2681859.html'
        },
        {
            brand: 'asus',
            model: 'tuf oc',
            series: '3080',
            url: 'https://www.mediamarkt.de/de/product/-2681861.html'
        },
        {
            brand: 'gigabyte',
            model: 'eagle oc',
            series: '3080',
            url: 'https://www.mediamarkt.de/de/product/-2683942.html'
        },
        {
            brand: 'gigabyte',
            model: 'gaming oc',
            series: '3080',
            url: 'https://www.mediamarkt.de/de/product/-2683937.html'
        },
        {
            brand: 'inno3d',
            model: 'ichill x3',
            series: '3080',
            url: 'https://www.mediamarkt.de/de/product/-2684241.html'
        },
        {
            brand: 'inno3d',
            model: 'ichill x4',
            series: '3080',
            url: 'https://www.mediamarkt.de/de/product/-2684238.html'
        },
        {
            brand: 'msi',
            model: 'gaming x trio',
            series: '3080',
            url: 'https://www.mediamarkt.de/de/product/-2683227.html'
        },
        {
            brand: 'msi',
            model: 'ventus 3x oc',
            series: '3080',
            url: 'https://www.mediamarkt.de/de/product/-2683229.html'
        },
        {
            brand: 'zotac',
            model: 'trinity',
            series: '3080',
            url: 'https://www.mediamarkt.de/de/product/-2683243.html'
        },
        {
            brand: 'asus',
            model: 'strix',
            series: '3090',
            url: 'https://www.mediamarkt.de/de/product/-2681863.html'
        },
        {
            brand: 'asus',
            model: 'strix oc',
            series: '3090',
            url: 'https://www.mediamarkt.de/de/product/-2681866.html'
        },
        {
            brand: 'asus',
            model: 'tuf',
            series: '3090',
            url: 'https://www.mediamarkt.de/de/product/-2681855.html'
        },
        {
            brand: 'asus',
            model: 'tuf oc',
            series: '3090',
            url: 'https://www.mediamarkt.de/de/product/-2681857.html'
        },
        {
            brand: 'gigabyte',
            model: 'aorus master',
            series: '3090',
            url: 'https://www.mediamarkt.de/de/product/-2691441.html'
        },
        {
            brand: 'gigabyte',
            model: 'aorus xtreme',
            series: '3090',
            url: 'https://www.mediamarkt.de/de/product/-2691440.html'
        },
        {
            brand: 'inno3d',
            model: 'ichill x4',
            series: '3090',
            url: 'https://www.mediamarkt.de/de/product/-2684235.html'
        },
        {
            brand: 'msi',
            model: 'gaming x trio',
            series: '3090',
            url: 'https://www.mediamarkt.de/de/product/-2683226.html'
        },
        {
            brand: 'sony',
            model: 'ps5 console',
            series: 'sonyps5c',
            url: 'https://www.mediamarkt.de/de/product/-2661938.html'
        },
        {
            brand: 'sony',
            model: 'ps5 digital',
            series: 'sonyps5de',
            url: 'https://www.mediamarkt.de/de/product/-2661939.html'
        },
        {
            brand: 'microsoft',
            model: 'xbox series x',
            series: 'xboxsx',
            url: 'https://www.mediamarkt.de/de/product/-2677360.html'
        },
        {
            brand: 'microsoft',
            model: 'xbox series s',
            series: 'xboxss',
            url: 'https://www.mediamarkt.de/de/product/-2677359.html'
        }
    ],
    name: 'mediamarkt'
};

dotenv file:

ASCII_BANNER=
ASCII_COLOR=
AUTO_ADD_TO_CART=
BROWSER_TRUSTED=
COUNTRY=
DESKTOP_NOTIFICATIONS=
DISCORD_NOTIFY_GROUP=
DISCORD_NOTIFY_GROUP_3060TI=
DISCORD_NOTIFY_GROUP_3070=
DISCORD_NOTIFY_GROUP_3080=
DISCORD_NOTIFY_GROUP_3090=
DISCORD_NOTIFY_GROUP_CORSAIR_SF=
DISCORD_NOTIFY_GROUP_RX6800=
DISCORD_NOTIFY_GROUP_RX6800XT=
DISCORD_NOTIFY_GROUP_RX6900XT=
DISCORD_NOTIFY_GROUP_RYZEN5600=
DISCORD_NOTIFY_GROUP_RYZEN5800=
DISCORD_NOTIFY_GROUP_RYZEN5900=
DISCORD_NOTIFY_GROUP_RYZEN5950=
DISCORD_NOTIFY_GROUP_SONYPS5C=
DISCORD_NOTIFY_GROUP_SONYPS5DE=
DISCORD_NOTIFY_GROUP_XBOXSX=
DISCORD_NOTIFY_GROUP_XBOXSS=
DISCORD_NOTIFY_GROUP_TEST=
DISCORD_WEB_HOOK=
EMAIL_PASSWORD=
EMAIL_TO=
EMAIL_USERNAME=
HEADLESS=
IN_STOCK_WAIT_TIME=
LOG_LEVEL=
LOW_BANDWIDTH=
MAX_PRICE_SERIES_3060TI=
MAX_PRICE_SERIES_3070=
MAX_PRICE_SERIES_3080=
MAX_PRICE_SERIES_3090=
MAX_PRICE_SERIES_CORSAIR_SF=
MAX_PRICE_SERIES_RX6800=
MAX_PRICE_SERIES_RX6800XT=
MAX_PRICE_SERIES_RX6900XT=
MAX_PRICE_SERIES_RYZEN5600=
MAX_PRICE_SERIES_RYZEN5800=
MAX_PRICE_SERIES_RYZEN5900=
MAX_PRICE_SERIES_RYZEN5950=
MAX_PRICE_SERIES_SONYPS5C=
MAX_PRICE_SERIES_SONYPS5DE=
MAX_PRICE_SERIES_XBOXSS=
MAX_PRICE_SERIES_XBOXSX=
MAX_PRICE_SERIES_TEST=
MICROCENTER_LOCATION=
MQTT_BROKER_ADDRESS=
MQTT_BROKER_PORT=
MQTT_CLIENT_ID=
MQTT_PASSWORD=
MQTT_QOS=
MQTT_TOPIC=
MQTT_USERNAME=
NVIDIA_ADD_TO_CART_ATTEMPTS=
NVIDIA_SESSION_TTL=
OPEN_BROWSER=false
PAGE_BACKOFF_MIN=10000
PAGE_BACKOFF_MAX=600000
PAGE_SLEEP_MIN=
PAGE_SLEEP_MAX=
PAGE_TIMEOUT=50000
PAGERDUTY_INTEGRATION_KEY=
PAGERDUTY_SEVERITY=
PHONE_CARRIER=
PHONE_NUMBER=
PLAY_SOUND=
PROXY_ADDRESS=
PROXY_PROTOCOL=
PROXY_PORT=
PUSHBULLET=
PUSHOVER_EXPIRE=
PUSHOVER_RETRY=
PUSHOVER_TOKEN=
PUSHOVER_USER=
PUSHOVER_PRIORITY=
SCREENSHOT=
SHOW_ONLY_BRANDS=
SHOW_ONLY_MODELS=
SHOW_ONLY_SERIES=sonyps5c,sonyps5de,xboxsx
SLACK_CHANNEL=
SLACK_TOKEN=
SMTP_ADDRESS=
SMTP_PORT=
STORES=amazon-de:50000:80000,alternate:50000:80000,amazon-de-warehouse:50000:80000,amd-de:30000:80000,asus-de:50000:80000,caseking:50000:80000,mediamarkt:100000:150000,medimax:50000:80000,mindfactory:50000:80000,otto:50000:80000,pcking:50000:80000,saturn:100000:300000
WEB_PORT=

Logs

selop commented 3 years ago

There is already an issue for this https://github.com/jef/streetmerchant/issues/1545. And there is a PR which fixes the issue - https://github.com/jef/streetmerchant/pull/1549

Lars4k commented 3 years ago

There is already an issue for this #1545. And there is a PR which fixes the issue - #1549

As you can See the fix is already in place. I quoted mediamarkt.ts above. But the fix Do not solve my 403 error.

wofnull commented 3 years ago

No timing out on my end, seams to be an temporary issue since http error code 403 means forbidden, it could be that you request to often the two shops causing them to start a bot protection against hammering the server over and over again.

[19:11:25] info :: ✖ [saturn] [asus (3080)] tuf :: OUT OF STOCK [19:11:25] info :: ✖ [mediamarkt] [asus (3080)] tuf :: OUT OF STOCK [19:11:28] info :: ✖ [saturn] [asus (3080)] tuf oc :: OUT OF STOCK [19:11:28] info :: ✖ [mediamarkt] [asus (3080)] tuf oc :: OUT OF STOCK

selop commented 3 years ago

@Lars4k Sorry I mixed up the title description.

I see that you use mediamarkt:100000:150000 in your config. With these timings you seem to still run into the backoff mode which then returns 403 for a given link for a while. At least for the PS5.

I tuned my config for mediamarkt with these timings and it seems to work since then - mediamarkt:15000:30000. But consider that this only applies for the GPU cards. I think it is very likely that they have harsher limiting for the Playstation 5 for obvious reasons.

Lars4k commented 3 years ago

Thanks guys. This is very strange, I also tried to use some proxys without success - 403 remains in place.

I will try to set the timings even higher.

EliasSHH commented 3 years ago

I have the same issue when trying to access amd-ca, I get the error 403. I've tried changing my Page_backoff and Page_sleep values, but to no avail.

github-actions[bot] commented 3 years ago

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days

github-actions[bot] commented 3 years ago

This issue has been closed because it is stale. Reopen if necessary.