[BUG] Bot fails to detect available flats on wbm.de after recent update

[xkrxkrxkr]

Describe the bug I've encountered an issue today where the bot has stopped identifying available flats on wbm.de, despite there being 5 flats currently listed (as per the archived page). This morning, the bot was functioning correctly; it detected a WBS flat (which was excluded based on my configuration) and attempted to apply for a non-WBS flat but was unsuccessful due to an issue. Following a fix issued by @Vel-San, I updated the Docker image, and since then, the bot has not detected any available flats.

To Reproduce

Expected behavior The bot should consistently detect available flats on wbm.de, including after updates and fixes are applied.

Screenshots

Environment Digest: sha256:a989a4dfe0a7817060d906620329f294f641ecaa8006970f94b60edf7b84745a

Additional context Not a coding expert, please let me know if there's any additional information I can share to help solving this.

[Vel-San]

Hi @xkrxkrxkr and welcome to Github!

Thanks for reporting this.

Can you confirm, that as of now, the bot is not finding any listings for you even tho it is showing on WBM?

https://www.wbm.de/wohnungen-berlin/angebote/

I see at least 4 listings now, and the bot on my side can find them just fine (And ignores because of my filters). Please let me know and make sure the image that you are using is not an outdated image, after you've pulled it.

I will add a "Bot Version" to the log in a future hotfix to always make sure which version you currently have.

My current log (In this very minute):

[xkrxkrxkr]

Hi @Vel-San,

thank you for your speedy response. Unfortunately it's still not detecting available flats. Please see my log:

[Vel-San]

@xkrxkrxkr

You might be using an older tag instead of latest. Can you please delete all your images:

docker rmi

Then pull the latest one image and try again (Below command will automatically pull the image):

Replace password and path with your own.

docker run -d \
    -e "EMAIL_PASSWORD=XXX" \
    -v <YOUR_PATH>/offline_viewings:/home/offline_viewings \
    -v <YOUR_PATH>/logging:/home/logging \
    -v <YOUR_PATH>/configs:/home/configs \
    vel7an/wbmbot_v2:latest

[Vel-San]

@xkrxkrxkr

Note: First run with -it flag means interactive so you can setup the config. If you already have your config, from now on you can simply either update it manually (Local file) to update, or just keep running the docker image with -d flag. You can see the logs and so on within the docker desktop app itself.

[xkrxkrxkr]

@Vel-San

Thanks so much for your help so far. As instructed, I deleted all my images and pulled the latest one.

I could see that there was a flat available just now on wbm.de, but for some reason the bot is still not detecting it.

[Vel-San]

@xkrxkrxkr

Okay, there is definitly something weird going on because I can see the current flat in my logs

Are you running this on windows btw? I am running this on Ubuntu/Linux. I am not sure if the dockerized bot acts differently on different platforms.

If you can run the bot directly (Without docker) please do that and let me know if it makes any difference. I just pushed some changes that add versioning to the bot (And to the docker image), along with tags/change-log releases inside the repo.

This should help me figure things out per version.

Docker Tags: https://hub.docker.com/r/vel7an/wbmbot_v2/tags Releases: https://github.com/Vel-San/wbmbot_v2/releases/

Also note the (version) in the first message of the bot's log.

[xkrxkrxkr]

@Vel-San

I'm on Amazon Linux. Again, I removed all the existing docker images, and pulled the most recent one. According to the log it's v1.0.0.

I'll try running the bot without docker tomorrow. In the meantime, could you kindly let me know in case there will be any flats available so I can compare with my log?

Thank you again for your support.

[Vel-San]

@xkrxkrxkr

Can you make sure that the entry point is the same in your launch commands?

https://github.com/Vel-San/wbmbot_v2/blob/3d00cd0600ab67ddac6631aa0fd0dfe757c60f02/ci/docker/Dockerfile#L41

Reason is that in Docker I think the bot acts differently if you are not running it as -H (headless). I can add a message to make sure that it is running in headless mode (will do it shortly).

If you are able to run it on your local machine (Docker Desktop) that would also be nice. (On Windows you can use WSL2 to have a linux environment).

Right now, there are no flats at all, I will notify here once there is anything!

[xkrxkrxkr]

@Vel-San

I just tried running the script via Docker Desktop on Mac. I simply pulled the image, and then clicked on Run. I received the following error, however.

I will go over your other comments, and post any feedback.

[Vel-San]

@xkrxkrxkr

EDIT: I just noticed you are running it on MacOS - I had so many problems with Docker not working properly on MacOS vs Linux/Windows, you need to enable extra settings from your OS and within Docker I think

I was about to come and ping you as I saw some flats (WBS) right now.

My Docker settings are the following:

And I run it from WSL2 terminal (Which shows in Docker Desktop afterwards) using the command I pasted above.

[xkrxkrxkr]

Hi @Vel-San

Yep, indeed, I'm on MacOS. Unfortunately my Docker Desktop settings look different. I don't have access to a Windows machine right now. I'll see if I have any luck with your suggestions. Thanks again for your help!

Re: Dockerfile on Amazon Linux, it looks the same to me.

[Vel-San]

@xkrxkrxkr

EDIT: I will build a new image right now with selenium as base instead of Ubuntu, hopefully that fixes your issues on MacOS, give me a few minutes.

Can you provide further information about your OS and Machine? (e.g. Which MacOS do you have? And which Laptop is it? Does it have any new chips like M1 M2 etc.. ?)

I see some problems from a quick google search related to Docker containers and Selenium under MacOS.

I had some problems with another container that I made sometime ago, and these fixes at least worked:

1. In your Docker settings, disable/uncheck Use gRPC FUSE for file sharing and hit Restart to Apply changes
2. If the above doesn't solve your issue, go to your Machine Settings > Security & Privacy > Give Full Disk Usage to Docker.app (If this still doesn't work, give your terminal as well)

And try again.

We can also try if you build the image directly on your machine instead and see if everything works fine, but for that, you'll have to clone or download the repo.

[xkrxkrxkr]

@Vel-San

I'm on MacOS 12.5.1, MacBook Pro 14-inch, 2021, Chip: Apple M1 Pro.

I also tried installing the script on Amazon Linux and running it without Docker, but it unfortunately didn't work. :(

[Vel-San]

@xkrxkrxkr

It has to be "Headless" on Amazon Linux too, so you should use -H!

Try this image please:

docker pull vel7an/wbmbot_v2:macos_test

I made it using the official selenium base instead of Ubuntu. If this still fails, can you please clone the repo (Or just download it) and build the image yourself? Just to see if building on MacOS makes any difference.

[xkrxkrxkr]

@Vel-San

OK, I see. Thank you! Unfortunately, it's still not detecting flats on Amazon Linux.

Thank you for providing the new macos_test image. Unfortunately it's the same result as before.

I'll look into how I can build the image myself. Thanks again for your support.

[Vel-San]

@xkrxkrxkr

Ok, let us try a couple of things now:

on Amazon Linux

First, install the dependecies:

apt-get update && \
    apt-get install -y --no-install-recommends \
    python3 \
    python3-pip \
    nano \
    wget \
    gnupg \
    xvfb

Second, try to install google chrome stable:

You can copy all of this and run in a single command:

wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add - && \
echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google-chrome.list && \
apt-get update && \
apt-get install -y google-chrome-stable

And try to run the tool again. If that doesn't work, try to install the debian yourself:

1-by-1

wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
dpkg -i google-chrome-stable_current_amd64.deb
apt-get -f install

If all above fails, please try to build the image yourself:

-Clone/Download the repo on your Amazon Linux -Build the Container as per the instructions in the README

Once we debug this and isolate the Amazon Linux issue, we can try and see why the MacOS related things aren't working.

[xkrxkrxkr]

OK, great. I tried to follow these steps for Amazon Linux and consulted with ChatGPT, since the commands didn't work. According to ChatGPT, this was the reason:

The command you were asked to run uses apt-get, which is a package management command used in Debian and Ubuntu distributions of Linux. However, since you are on Amazon Linux, which is based on the Red Hat Enterprise Linux (RHEL) distribution, the equivalent package manager is yum or dnf for more recent versions. Here’s how you can adapt the command to work on Amazon Linux:

I did go ahead with the following command for step 1, as per ChatGPT suggestion. sudo yum install -y python3 python3-pip nano wget gnupg2 Xvfb

Here's the result

sudo yum install -y python3 python3-pip nano wget gnupg2 Xvfb google-chrome 25 kB/s | 3.6 kB 00:00
Package python3-3.9.16-1.amzn2023.0.6.x86_64 is already installed. Package python3-pip-21.3.1-2.amzn2023.0.7.noarch is already installed. Package nano-5.8-3.amzn2023.0.3.x86_64 is already installed. Package wget-1.21.3-1.amzn2023.0.3.x86_64 is already installed. Error: Problem: problem with installed package gnupg2-minimal-2.3.7-1.amzn2023.0.4.x86_64

• package gnupg2-minimal-2.3.7-1.amzn2023.0.4.x86_64 conflicts with gnupg2 provided by gnupg2-2.3.7-1.amzn2023.0.3.x86_64
  • package gnupg2-minimal-2.3.7-1.amzn2023.0.3.x86_64 conflicts with gnupg2 provided by gnupg2-2.3.7-1.amzn2023.0.3.x86_64
• conflicting requests
• package gnupg2-minimal-2.3.7-1.amzn2023.0.4.x86_64 conflicts with gnupg2 provided by gnupg2-2.3.7-> 1.amzn2023.0.4.x86_64
• package gnupg2-minimal-2.3.7-1.amzn2023.0.3.x86_64 conflicts with gnupg2 provided by gnupg2-2.3.7-1.amzn2023.0.4.x86_64 (try to add '--allowerasing' to command line to replace conflicting packages or '--skip-broken' to skip uninstallable packages)

I downloaded & installed Chrome using the following commands instead: sudo dnf install wget -y wget https://dl.google.com/linux/direct/google-chrome-stable_current_x86_64.rpm

sudo dnf localinstall google-chrome-stable_current_x86_64.rpm -y

followed by

google-chrome-stable --headless

Still not detecting flats.

I'll try to go ahead and build the image myself. I'll report back.

[Vel-San]

@xkrxkrxkr

Oh well :( We'll have to wait for the build results then.. I doubt it will fix it but given that I've built this and it works on both Linux WSL and from Docker Desktop connected to WSL, I didn't think of corner use-cases that might be problematic like Amazon Linux or MacOS x_x

Let me know what is the outcome of the builds then!

[xkrxkrxkr]

Looking back, I probably should've switched to Ubuntu sooner—it might have saved us a bunch of time. Even though there are no available flats on the wbm.de site right now, I feel like it will work properly now. I realized just now, that when the bot stopped working on my end, I didn't even get the usual success message about cookies being accepted. But guess what? For the first time, that message finally popped up:

[23.02.2024 - 15:49] [webDriverOperations.py:252] [SUCCESS] Cookies have been accepted 🍪.

[Vel-San]

@xkrxkrxkr

That looks better now, yes! Let me know once you are able to see any flats there.

It will take a lot of debugging to figure out what is wrong with the MacOS / Amazon Linux installations tho, so might not be worth to dig deeper there.

[xkrxkrxkr]

@Vel-San

Success! The bot works as it should on Ubuntu.

I appreciate the time you spent to help me fixing this. I hope you'll have a great weekend. Let me know if I should close this issue.

[Vel-San]

@xkrxkrxkr

Awesome stuff! So it was indeed the Selenium not working properly.

I'm glad its working now. Happy Hunting! We can close this until further notice.