Testing Docker

[mgrimace]

Hello, I figured I'd contribute by testing out the docker implementation. tl;dr - Initial run has no issues(!), testing scheduled runs next.

Here's the steps I've taken in case anyone more knowledgable can test/contribute/make suggestions.

System

Headless Debian 12 server, running on a Proxmox LXC

Steps

• Downloaded the source code
• Updated accounts.json
• Made sure config.json is set to "headless": true,
• Per #29 I also changed "searchDelay": { to "min": 180000, and "max": 270000 because I'm in a region with a cooldown.

Build and run the docker image

• Run docker build -t microsoft-rewards-script-docker . to build the image
• docker run --name netsky -d microsoft-rewards-script-docker omit the detached flag -d to view the script output in terminal. Optionally, change the name of the container.

Schedule daily runs

This is where I'm less certain as to best method to do this, presently:

• Create a daily.sh script, which should add a degree of randomness to the start time:

#!/bin/bash

MINWAIT=$((5*60))
MAXWAIT=$((50*60))
SLEEPTIME=$((MINWAIT+RANDOM % (MAXWAIT-MINWAIT)))
echo "sleeping for "$((SLEEPTIME / 60))"m ("$SLEEPTIME"s) ..." 
sleep $SLEEPTIME
echo "starting container"
docker start netsky

• Make sure the script is executable
• Set up cron to run the script daily using crontab -e and entering the following line, which should run the script at 5:30 am daily + the random sleeptime calculation above:

30 5 * * * /bin/bash /root/Microsoft-Rewards-Script/daily.sh

Results

I'll report back with results here. I'm more comfortable using docker compose, so this is new territory for me. Suggestions welcome.

[mgrimace]

Initial manual run of the script completed without error. I had already completed searches, etc. earlier in the day, so I can't say for certain if those would have worked, but no errors getting it up and running!

[mgrimace]

Ok, scheduled run started as expected. Desktop searches completed, and mobile started but ran into an error approximately half-way through:

[3/28/2024, 1:45:26 PM] [PID: 31] [LOG] [MAIN] Unable to complete mobile searches, bad User-Agent? Retrying...

It retried 5 times, with errors relating to timeout exceeded (I can provide additional logs, not sure what’s helpful), then stopped.

Edit: second run of the container completed the remaining searches, which, for me is more-or-less typical behaviour (I ran the base script 2x daily previously in case anything didnt’ finish)

[mgrimace]

Testing run via an early/rough compose.yaml with built-in scheduling, and definitely needs testing and revision. The goal is to keep the container running while the script itself is re-run daily per the in-built cron schedule and script. There is very likely a smarter/more efficient way to do this.

The following assumes you have a script called run_daily.sh located in the root of the package (/Microsoft-Rewards-Script/run_daily.sh). I'm testing out the following script, which is slightly changed from above to re-run the rewards script rather than restart the container.

run_daily.sh

#!/bin/bash

# Define the minimum and maximum wait times in seconds
MINWAIT=$((5*60))  # 5 minutes
MAXWAIT=$((50*60)) # 50 minutes

# Calculate a random sleep time within the specified range
SLEEPTIME=$((MINWAIT + RANDOM % (MAXWAIT - MINWAIT)))

# Convert the sleep time to minutes for logging
SLEEP_MINUTES=$((SLEEPTIME / 60))

# Log the sleep duration
echo "Sleeping for $SLEEP_MINUTES minutes ($SLEEPTIME seconds)..."

# Sleep for the calculated time
sleep $SLEEPTIME

# Log the start of the script
echo "Starting script..."

# Start the script
docker exec netsky npm run start

compose.yaml

services:
  microsoft-rewards-script:
    build: .
    container_name: netsky
    environment:
      - NODE_ENV=production
      - TZ=America/Toronto #change to your timezone
    volumes:
      - /path/to/Microsoft-Rewards-Script:/usr/src/microsoft-rewards-script #make sure this matches where you've installed the package
    restart: unless-stopped
  cron:
    image: alpine
    container_name: cron-container
    volumes:
      - /path/to/Microsoft-Rewards-Script:/usr/src/microsoft-rewards-script #should be same as above
    restart: unless-stopped
    command: ["sh", "-c", "echo '30 5,11 * * * root /usr/src/microsoft-rewards-script/run_daily.sh' > /etc/crontabs/root && crond -f"] # this runs script daily at 5:30 am and 11:30 am of your timezone using the added randomness from the run_daily.sh, feel free to change

[mgrimace]

Testing run via an early/rough compose.yaml with built-in scheduling, and definitely needs testing and revision. The goal is to keep the container running while the script itself is re-run daily per the in-built cron schedule and script. There is very likely a smarter/more efficient way to do this.

This setup won't work. The rewards script exits after running, then immediately restarts, so it'll run constantly. Not sure how to fix at the moment, but I'll keep testing. The docker build + docker run setup work fine in the meantime

[Ryely-Squires]

Following. I'm absolutely horrid with regular docker, I only use compose, and throwing this bot in a container would be nice.

[mgrimace]

Following. I'm absolutely horrid with regular docker, I only use compose, and throwing this bot in a container would be nice.

Ha me too! If you want test a compose, please try the one below. AFAIK, you'd want to place this as a compose.yaml the root of the Microsoft-Rewards-Script folder, and run with docker compose up -d or actually omit the -d to watch if it works.

If you have previously used the docker run commands, you may have to use docker stop netsky and docker rm netsky to remove the previously ran version.

services:
  microsoft-rewards-script:
    build: .
    container_name: netsky
    environment:
      - NODE_ENV=production
      - TZ=America/Toronto #customize, may not be necessary at the moment 
    volumes:
      - /path/to/Microsoft-Rewards-Script:/usr/src/microsoft-rewards-script #customize this to wherever you've installed the package

This should run once, then exit. For now, if you want to schedule it, copy the run_daily.sh below into the same folder, and add a cron entry using crontab -e and add the following line 30 5 * * * /bin/bash /path/to/Microsoft-Rewards-Script/daily.sh (making sure to actually use the real path, not the placeholder I have here). It's set for 5:30 am, if you want it to run twice, just do a comma after the 5 (e.g., 30 5,13 * * * for 5:30 am and 1:30 pm, or customize as you wish).

run_daily.sh

#!/bin/bash

MINWAIT=$((5*60))
MAXWAIT=$((50*60))
SLEEPTIME=$((MINWAIT+RANDOM % (MAXWAIT-MINWAIT)))
echo "sleeping for "$((SLEEPTIME / 60))"m ("$SLEEPTIME"s) ..." 
sleep $SLEEPTIME
echo "starting container"
docker start netsky

Finally, if you've been messing around like I have and building these left and right, you might want to run a docker system prune -a -f to cleanup.

[mgrimace]

The next goal would be to add environmental variable support for things like headless=true directly into the compose, and support for environmental secrets (.env) for account information, but that requires updating the dockerfile, configs, and possible other things that are outside my basic skills!

[Ryely-Squires]

I've been messing with the compose for a little while now, and I can get it to actually startup (along with the webhook!) but have been getting errors within the program itself. I'm sure they're on my end in terms of my setup (I recently transferred from GUI to CLI Debian, everything is a learning curve). But I can confirm that the compose works well, and shows logs! If I can get it fully working, I'll let you know. Great work.

[mgrimace]

I've been messing with the compose for a little while now, and I can get it to actually startup (along with the webhook!) but have been getting errors within the program itself. I'm sure they're on my end in terms of my setup (I recently transferred from GUI to CLI Debian, everything is a learning curve). But I can confirm that the compose works well, and shows logs! If I can get it fully working, I'll let you know. Great work.

Thanks for testing! Here is a simplified compose that should be run from the root directory:

services:
  microsoft-rewards-script:
    build: .
    container_name: netsky
    environment:
      - NODE_ENV=production
    volumes:
      - .:/usr/src/microsoft-rewards-script

AFAIK, sending logs/results to Discord via webhooks should not require incoming ports to be exposed and it should work out-of-the-box. Can't speak to other errors though, but check my other post #29 if searches are failing (due to cooldown requirements in some countries)

[Ryely-Squires]

The error was a network one (net::ERR_NETWORK_CHANGED) which seems to have gone away with redoing the setup process? I'm not sure, but the bot has ran successfully once.

I haven't been able to test whether it can correctly search for points yet because I have already received them for today, but I've put the shell script and edited cron accordingly. I will again update you tomorrow on whether or not it successfully runs automatically with the scripts.

[Ryely-Squires]

The script seems to work.

I should mention that the dreaded network error seems to have reared it's head, but whether or not it'll happen seems to be random with each restart? I'm not sure, I'll likely make a separate issue on the matter if I can't figure it out. I had to run the shell script manually a few times (with shortened durations) to get the container to actually function. And looking at the cronjob, it should be fine.

[mgrimace]

The script seems to work.

I should mention that the dreaded network error seems to have reared it's head, but whether or not it'll happen seems to be random with each restart? I'm not sure, I'll likely make a separate issue on the matter if I can't figure it out. I had to run the shell script manually a few times (with shortened durations) to get the container to actually function. And looking at the cronjob, it should be fine.

It’s possible it’s a docker-related networking error, try docker network prune if you’ve been playing around in docker. Here’s a similar issue on Stackoverflow (link)

[Ryely-Squires]

I have tried that command a few times in testing, I thought maybe because I had overdone it on creating containers/networks, but it seems to be the same. Regardless, running it a few times basically guarantees that at least one of them will function, so I have the cronjob entry modified for a few times rather than one just to account for that. Otherwise, the setup seems to work well.

[mgrimace]

Ok, if you need to rebuild the script after making changes to the config, you'll need to run docker compose up -d --build to rebuild or else it'll just re-up the existing/previous build.

If you've re-built the image a few times now, you'll also like want to run docker system prune -a -f to clean up all the builds/images.

[mgrimace]

I'm currently testing automatic scheduled daily runs via docker. If you can contribute or test, please take a look at my docker test branch: https://github.com/mgrimace/Microsoft-Rewards-Script/tree/docker

It should run the script on the initial container start; then, it'll keep the container idle, and re-run the script daily at 5:00 am.

The schedule is customizable in the crontab file found under /src/. use crontab.guru if you're not sure how to change cron syntax.

If it works, ideally the next step would be to be able to customize the value via an environments: argument in the docker compose file.

If you are able to help or contribute to docker implementation in any way, please feel free to add a PR to my testing branch (and once everything is confirmed working, I'll PR it to main here).

[Ryely-Squires]

Hey, SSH into my server isn't working, so if nobody follows up before I head over there for the weekend, I'd be happy to test it out (although the current cronjob implementation works just fine). This seems a lot more user friendly.

[mgrimace]

Hey, SSH into my server isn't working, so if nobody follows up before I head over there for the weekend, I'd be happy to test it out (although the current cronjob implementation works just fine). This seems a lot more user friendly.

Thanks so much, and no rush, I'm kind of learning as I go with this. My overall intention with docker is to have everything 'built in' to the service and not require any coding/scripting (because that's something I struggle with myself).

Out of curiosity, do you think the default implementation should be one run per day (e.g., 5 am)? Or two runs (e.g., 5 am, 11am) to catch anything that was missed? I run it twice myself

[Ryely-Squires]

Almost certainly twice per day; I even run it three times as a precaution. The script tends to miss things on the first run.

[mgrimace]

Almost certainly twice per day; I even run it three times as a precaution. The script tends to miss things on the first run.

Great ok! I'm testing an update at the moment that adds custom cron schedules that can be set via environmental variables in compose, or in the config.json. It is not yet live on my docker testing branch (I want to make sure in-built scheduling actually works before pushing this update, which could make things messier)

With environmental variable support, the default value is 0 5,11 * * * which is pre-set in the config.json, but any changes made in the compose file should take precedence. For example, you could run it three times daily with:

services:
  microsoft-rewards-script:
    build: .
    container_name: netsky
    environment:
      - NODE_ENV=production
      - CRON_START_TIME=0 5,11,16 * * *   #Set your preferred schedule here, use crontab.guru for proper cron syntax 
    volumes:
      - .:/usr/src/microsoft-rewards-script

If a user doesn't set it in the compose, it should still default to 5,11; however, I'm attempting to make sure that if cron is removed from the compose AND from the config.json that the script only runs on container launch (which is the base functionality). Testing it for a 24 hr cycle then I'll push that update.

Upcoming goal is to set other env variables like headless mode, search delays, etc. that can all be done in the compose.

[mgrimace]

I should mention this is not yet live in my docker branch. It only has the in-built scheduling, not the env variable support yet (I want to test it further before breaking the current in-built scheduling).

If you or anyone do test the in-built scheduling, remember to comment out your exising crontab entries by using crontab -e and andding a # before you current schedule. The container should schedule and run itself, and look for it to run the script on the next schedule (5am) automatically.

You can use docker logs netsky to see what the script is doing in the container.

[mgrimace]

Ok, I pushed a huge update to my docker testing branch.

This update includes substantial changes to the DOCKERFILE to enable docker environmental variable support for all config options. Absolutely no changes are made to the core .ts files or underlying script.

I added each env var to the config table on the main README.md. HEADLESS=true is the only mandatory one for docker.

Scheduling:

• You can use the var - CRON_START_TIME= in run or compose to set your own preferred schedule. The script will run automatically on container start, then stay idle until the next schedule run.
• Scheduling is built into the container and requires no user scripting or system cron entries, it is all done via env var in run or compose. If the user does not set a CRON_START_TIME, it will default to 5 am and 11 am local time.
• Remember to disable or comment out any previous cron schedules on your system via crontab -e.

Set config options via environment vars:

• All config options have corresponding env vars that can be used in compose or run.
• I added Timezone support for the container
• If specified, the custom env var value will be prioritized over the default values present in the config.json. As in, you do not need to change the values in the config.json if you set a custom env var. For example, if you have HEADLESS=false in the config.json (default value), but set it as =true in the compose.yaml, it will run as HEADLESS=true.
• If you do not specify a particular env var, the script will automatically use the default value present in the config.json.

I'm learning as I go and doing this in my spare time. There will likely be bugs and things I've missed. Please be patient, and test and contribute wherever possible. PRs are welcome and encouraged.

https://github.com/mgrimace/Microsoft-Rewards-Script/tree/docker

I'll PR this back to main here after preliminary testing.

[mgrimace]

Ok, tl;dr - the docker container runs the script daily as scheduled by the user in compose/run command. It should have full support for env variables, which are prioritized over the config. This means, you can (hopefully) do everything except your account from a compose file or run command.

I removed running the script on container start because it was getting impossible to test. It only runs as scheduled with a default of 5 and 11, customizable by CRON_START_TIME. Kept script which adds randomness to scheduled start times as a preventative measure.

I've pushed a few updates to my docker testing branch to fix scheduling and timezone.

Next steps - account support for docker compose.yaml, maybe a 'run-once' env option, and exploring apprise support for better notifications (but this is tricky for me because multiple .ts files would need edititing). Further testing would be appreciated, PRs welcome. AFAIK this should be safe to push back here to main but I'm going to test a little more.

[Ryely-Squires]

Hey, sorry for taking so long.

I couldn't get it to run. The container on run shows timestamp of a time it currently isn't, and does so immediately (without the delay) going up to "Password entered successfully", and nothing after.

I've been troubleshooting for a while now, can't seem to fix it. My only guess is that it's conflicting with the original branch's installation, considering the time it says it's running at (1 AM) is what the main branch ran at.

I'm sure it'd probably work on a clean installation.

[mgrimace]

Hey, sorry for taking so long.

I couldn't get it to run. The container on run shows timestamp of a time it currently isn't, and does so immediately (without the delay) going up to "Password entered successfully", and nothing after.

I've been troubleshooting for a while now, can't seem to fix it. My only guess is that it's conflicting with the original branch's installation, considering the time it says it's running at (1 AM) is what the main branch ran at.

I'm sure it'd probably work on a clean installation.

No need to apologize at all, I appreciate you trying it out! I pushed an update that should fix the time in the container if you specify the timezone in the compose, but if it doesn’t work I’ll have to go back to the drawing board on that one.

On starting the container now, it shouldn’t do anything until the next scheduled run - so it probably won’t do or say anything at all in docker logs netsky until your first cron time. Even with the wrong time zone showing, it should run at scheduled actual time.

I disabled the ‘run on start’ because I couldn’t test the scheduling because it kept running each time I recreated the container.

Let me know if you’d prefer it to run immediately or keep it automatic/scheduled. I might try and introduce a ‘run once’ or ‘run on container start’ env option if I am able.

Thanks so much for testing!

[mgrimace]

quick follow-up ok the docker container date should be correct, but I see the script itself is using a different timezone in my run - I’ll follow-up. In the meantime cron will/should start at the proper local time, so it shouldn’t impact function.

You can check the container date with docker exec -it netsky date. Mine is showing the correct timezone but the script is running and showing the wrong time.

[Ryely-Squires]

The run on container start option would be useful. It'd make troubleshooting a ton easier, as I'm having trouble with getting the script to run. The date is correct on the container.

[mgrimace]

The run on container start option would be useful. It'd make troubleshooting a ton easier, as I'm having trouble with getting the script to run. The date is correct on the container.

Ok I'll do that and try and get it up today (or revert to previous behaviour if I can't make it run properly with the var). In the meantime, just in case when you rebuild a container:

docker stop netsky && docker rm netsky
docker system prune -a -f
docker compose up -d --build

the first two lines should fully stop, remove, and cleanup any leftovers from previous builds, the third line will rebuild using your newest dockerfile. Note the system prune command will remove untagged networks and other images that aren't used/running

[mgrimace]

@Ryely-Squires the latest update (April30) on my test should fix the following:

• The script should again, by default, run on container start, then continue per schedule (tested briefly, working on my end).
• This defaults to true; however, can be optionally over-ridden in the compose.yaml with the env var RUN_ON_START=false. If set to false, the script should only run on the specified cron schedule (untested).
• I also passed the TZ right into the npm start command, rather than just the container overall so it should show the proper time (tested; shows up proper on my end). Haven't tested whether TZ shows properly in the script on a scheduled run - I'll likely have to fix that too, but this should give you the basic functionality now.

If testing, please do the following to remove previous test containers, remove leftover images, and fully rebuild the container:

docker stop netsky && docker rm netsky
docker system prune -a -f
docker compose up -d --build

[Ryely-Squires]

Hey, sorry. Back at the machine.

The timezone is proper on the script when ran manually. But it still gets stuck at "Password entered successfully" while the main branch's script seems to work.

Specifically:

[5/4/2024, 8:19:55 PM] [PID: 25] [LOG] [MAIN] Bot started with 1 clusters [5/4/2024, 8:19:55 PM] [PID: 25] [LOG] [MAIN-WORKER] Started tasks for account XXXXXXXXXX (censored) [5/4/2024, 8:19:55 PM] [PID: 25] [LOG] [BROWSER] Created browser with User-Agent: "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36 Edg/122.0.0.0" [5/4/2024, 8:19:55 PM] [PID: 25] [LOG] [MAIN] Starting DESKTOP browser [5/4/2024, 8:20:08 PM] [PID: 25] [LOG] [LOGIN] Email entered successfully [5/4/2024, 8:20:11 PM] [PID: 25] [LOG] [LOGIN] Password entered successfully

Any ideas?

[mgrimace]

Any ideas?

AFAIK this is a 2FA issue.

If you have 2FA enabled, there's no way (at the moment) that I'm aware of to enter the code in headless mode (which isn't advised for docker as it is) - the easiest solution would be to disable 2FA on your account temporarily. As a first step, you could possible run the script manually without docker using headless=false and enter the login code into the browser manually to (hopefully) save the credentials, then rebuild/re-run docker with that fresh config folder. I'm not certain that will work, in fact, give that you can login when it's run manually I'll assume it's not working this way.

See issues #62 and #23 - as it is, the script doesn't have a way (AFAIK) to pass the 2FA code in headless mode at all (so at the moment I don't have a way when it's containerized)

Tl;dr try disabling 2FA on your rewards account and rerunning container

[OrangeChu]

I downloaded the source code, changed accounts. json, ran docker-compose up -d --build, and got the following error:

Current default time zone: 'Etc/UTC'
Local time is now:      Thu May 23 00:41:11 UTC 2024.
Universal Time is now:  Thu May 23 00:41:11 UTC 2024.

> microsoft-rewards-script@1.4.6 start
> node ./dist/index.js

node:internal/modules/cjs/loader:1143
  throw err;
  ^

Error: Cannot find module '/usr/src/microsoft-rewards-script/dist/index.js'
    at Module._resolveFilename (node:internal/modules/cjs/loader:1140:15)
    at Module._load (node:internal/modules/cjs/loader:981:27)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:128:12)
    at node:internal/main/run_main_module:28:49 {
  code: 'MODULE_NOT_FOUND',
  requireStack: []
}

And the TZ in compose.yaml is not effective? I changed to - TZ=Asia/Shanghai, but the log still shows TZ as Etc/UTC.

What did I miss?

[mgrimace]

I downloaded the source code, changed accounts. json, ran docker-compose up -d --build, and got the following error:

Current default time zone: 'Etc/UTC'
Local time is now:      Thu May 23 00:41:11 UTC 2024.
Universal Time is now:  Thu May 23 00:41:11 UTC 2024.

> microsoft-rewards-script@1.4.6 start
> node ./dist/index.js

node:internal/modules/cjs/loader:1143
  throw err;
  ^

Error: Cannot find module '/usr/src/microsoft-rewards-script/dist/index.js'
    at Module._resolveFilename (node:internal/modules/cjs/loader:1140:15)
    at Module._load (node:internal/modules/cjs/loader:981:27)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:128:12)
    at node:internal/main/run_main_module:28:49 {
  code: 'MODULE_NOT_FOUND',
  requireStack: []
}

And the TZ in compose.yaml is not effective? I changed to - TZ=Asia/Shanghai, but the log still shows TZ as Etc/UTC.

What did I miss?

hmm, it looks like you may have to manually build the script before creating it via docker. I missed that in my testing because I had it as a script already.

try cd path/to/Microsft-Rewards-Script/ then,

npm i
npx playwright install
npx playwright install-deps
npm run build

Then, docker compose up -d —build

Once the script is built then the dockerfile can pull all the /src/ stuff to build the image, your environmental vars (e.g., tz) should work.

I’ll have to see if this is something I introduced in my updates to the dockerfile or a quirk from the original.

Please let me know if my steps above help

[Ryely-Squires]

Was about to comment above lol, this is most certainly it (it gave me a bit of trouble when I was doing it).

Dist folder and its contents (including index.js) won't exist until manually built.

[OrangeChu]

I downloaded the source code, changed accounts. json, ran docker-compose up -d --build, and got the following

Current default time zone: 'Etc/UTC'
Local time is now:      Thu May 23 00:41:11 UTC 2024.
Universal Time is now:  Thu May 23 00:41:11 UTC 2024.

> microsoft-rewards-script@1.4.6 start
> node ./dist/index.js

node:internal/modules/cjs/loader:1143
  throw err;
  ^

Error: Cannot find module '/usr/src/microsoft-rewards-script/dist/index.js'
    at Module._resolveFilename (node:internal/modules/cjs/loader:1140:15)
    at Module._load (node:internal/modules/cjs/loader:981:27)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:128:12)
    at node:internal/main/run_main_module:28:49 {
  code: 'MODULE_NOT_FOUND',
  requireStack: []
}

And the TZ in compose.yaml is not effective? I changed to - TZ=Asia/Shanghai, but the log still shows TZ as Etc/UTC. What did I miss?

hmm, it looks like you may have to manually build the script before creating it via docker. I missed that in my testing because I had it as a script already.

try cd path/to/Microsft-Rewards-Script/ then,

npm i
npx playwright install
npx playwright install-deps
npm run build

Then, docker compose up -d —build

Once the script is built then the dockerfile can pull all the /src/ stuff to build the image, your environmental vars (e.g., tz) should work.

I’ll have to see if this is something I introduced in my updates to the dockerfile or a quirk from the original.

Please let me know if my steps above help

A new error has occurred

Current default time zone: 'Etc/UTC'
Local time is now:      Thu May 23 03:38:06 UTC 2024.
Universal Time is now:  Thu May 23 03:38:06 UTC 2024.

> microsoft-rewards-script@1.4.6 start
> node ./dist/index.js

node:internal/modules/cjs/loader:1143
  throw err;
  ^

Error: Cannot find module 'playwright'
Require stack:
- /usr/src/microsoft-rewards-script/dist/browser/Browser.js
- /usr/src/microsoft-rewards-script/dist/index.js
    at Module._resolveFilename (node:internal/modules/cjs/loader:1140:15)
    at Module._load (node:internal/modules/cjs/loader:981:27)
    at Module.require (node:internal/modules/cjs/loader:1231:19)
    at require (node:internal/modules/helpers:177:18)
    at Object.<anonymous> (/usr/src/microsoft-rewards-script/dist/browser/Browser.js:6:38)
    at Module._compile (node:internal/modules/cjs/loader:1364:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1422:10)
    at Module.load (node:internal/modules/cjs/loader:1203:32)
    at Module._load (node:internal/modules/cjs/loader:1019:12)
    at Module.require (node:internal/modules/cjs/loader:1231:19) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [
    '/usr/src/microsoft-rewards-script/dist/browser/Browser.js',
    '/usr/src/microsoft-rewards-script/dist/index.js'
  ]
}

Node.js v18.20.3

Why rely on the local environment if using Docker? As far as I know of Docker, it should include all running environments without requiring users to execute additional commands such as npm i?

I will still encounter some other problems. Can package it into an image and push it to https://hub.docker.com/?

[mgrimace]

A new error has occurred

Try npx playwright install and npx playwright install-deps after the npm i step.

Why rely on the local environment if using Docker? As far as I know of Docker, it should include all running environments without requiring users to execute additional commands such as npm i?

Yes, absolutely true, see my comments below. As far as I know it would require some changes to the script to allow the accounts to be added after the script is built.

I will still encounter some other problems. Can package it into an image and push it to https://hub.docker.com/?

100% agree, that would be my ideal implementation and up to @TheNetsky. As far as I know configs and accounts are built into the script each time it’s built.

A pre-built image would likely have to be the defaults/sample, and then have the ability to pass through config options (which I attempted to add via my recent PR), and pass through the account(s) which I haven’t done.

Ideally both config.json and accounts.json could be volume mapped to the container (e.g., /appdata/netsky/config), and AFAIK it would require the script to be updated to be accept those values after the base script/image is built.

[mgrimace]

oh @OrangeChu I should also mention that my latest PR that included tz support, env vars, etc., are not in the latest 1.4.6 release - probably the easiest thing to do is git clone the repo.

[mgrimace]

Ok, retested the original dockerfile and yes it does need the directories created by originally running the npm run build command locally (pfew, it's not a new bug that I introduced). This is not ideal for docker, as you mentioned it should all be entirely containerized. I'll see if this is something I can do at the Dockerfile level as I don't want to mess with the core script.

[OrangeChu]

A new error has occurred

Try npx playwright install and npx playwright install-deps after the npm i step.

Why rely on the local environment if using Docker? As far as I know of Docker, it should include all running environments without requiring users to execute additional commands such as npm i?

Yes, absolutely true, see my comments below. As far as I know it would require some changes to the script to allow the accounts to be added after the script is built.

I will still encounter some other problems. Can package it into an image and push it to https://hub.docker.com/?

100% agree, that would be my ideal implementation and up to @TheNetsky. As far as I know configs and accounts are built into the script each time it’s built.

A pre-built image would likely have to be the defaults/sample, and then have the ability to pass through config options (which I attempted to add via my recent PR), and pass through the account(s) which I haven’t done.

Ideally both config.json and accounts.json could be volume mapped to the container (e.g., /appdata/netsky/config), and AFAIK it would require the script to be updated to be accept those values after the base script/image is built.

After executing npx playwright install and npx playwright install deps, it can run normally. Thank you for your help.

I see that the dockerfile also includes the execution of npx playwright install and npx playwright install deps. Why do we need to execute these in the local environment to use them? I haven't used playwright in Docker before, I didn't expect it to be so troublesome.

[OrangeChu]

https://playwright.dev/docs/docker The Playwright package/dependency is not included in the image and should be installed separately.

I reviewed the Docker instructions for Playwright. It seems we have already installed the relevant dependencies in the Dockerfile. Strangely, why do we need to install Playwright dependencies locally to run Docker here?

Because I am installing this service on a Synology NAS, which is a bit different from other Linux systems. The system space allocated is quite small, leading to a "no space" error when installing Playwright. This requires a lot of extra work. If all the necessary environments could be included in a image, it would greatly improve the user experience.

[mgrimace]

https://playwright.dev/docs/docker The Playwright package/dependency is not included in the image and should be installed separately.

I reviewed the Docker instructions for Playwright. It seems we have already installed the relevant dependencies in the Dockerfile. Strangely, why do we need to install Playwright dependencies locally to run Docker here?

Because I am installing this service on a Synology NAS, which is a bit different from other Linux systems. The system space allocated is quite small, leading to a "no space" error when installing Playwright. This requires a lot of extra work. If all the necessary environments could be included in a image, it would greatly improve the user experience.

I think it’s an easy short term fix to include an npm run build command in the dockerfile to not have to install npm/playwright/etc locally and build the script within the container from the source. I’ll see if I can put together a PR for that in the next few days.

I totally agree re. an image, but I’m not the maintainer of the script, just a user who wanted to contribute some basic docker improvements based on my limited skill. Please feel free to add a PR if you can help out too!

[OrangeChu]

https://playwright.dev/docs/docker The Playwright package/dependency is not included in the image and should be installed separately.

I reviewed the Docker instructions for Playwright. It seems we have already installed the relevant dependencies in the Dockerfile. Strangely, why do we need to install Playwright dependencies locally to run Docker here? Because I am installing this service on a Synology NAS, which is a bit different from other Linux systems. The system space allocated is quite small, leading to a "no space" error when installing Playwright. This requires a lot of extra work. If all the necessary environments could be included in a image, it would greatly improve the user experience.

I think it’s an easy short term fix to include an npm run build command in the dockerfile to not have to install npm/playwright/etc locally and build the script within the container from the source. I’ll see if I can put together a PR for that in the next few days.

I totally agree re. an image, but I’m not the maintainer of the script, just a user who wanted to contribute some basic docker improvements based on my limited skill. Please feel free to add a PR if you can help out too!

Sincerely appreciate your efforts on this project!

[mgrimace]

https://playwright.dev/docs/docker The Playwright package/dependency is not included in the image and should be installed separately.

I reviewed the Docker instructions for Playwright. It seems we have already installed the relevant dependencies in the Dockerfile. Strangely, why do we need to install Playwright dependencies locally to run Docker here? Because I am installing this service on a Synology NAS, which is a bit different from other Linux systems. The system space allocated is quite small, leading to a "no space" error when installing Playwright. This requires a lot of extra work. If all the necessary environments could be included in a image, it would greatly improve the user experience.

I think it’s an easy short term fix to include an npm run build command in the dockerfile to not have to install npm/playwright/etc locally and build the script within the container from the source. I’ll see if I can put together a PR for that in the next few days. I totally agree re. an image, but I’m not the maintainer of the script, just a user who wanted to contribute some basic docker improvements based on my limited skill. Please feel free to add a PR if you can help out too!

Sincerely appreciate your efforts on this project!

Thanks, I'm learning as I go here with docker (I'm a docker user, but not a developer) and will happily pass things over if someone more experienced can jump in.

In the meantime, I pushed a change to my docker test branch that should automaticallly build the script entirely within docker - https://github.com/mgrimace/Microsoft-Rewards-Script/tree/docker

If you (or anyone) is willing to test:

1. Remove /node_modules and /dist from your Microsoft-Rewards-Script folder (which is where the script was 'built' locally)
2. Remove existing containers and images: docker stop netsky && docker rm netsky && docker system prune -a -f
3. Optionally, remove node.js/previous dependencies from your system
4. Clone the repo if you hadn't already
5. Use the updated compose.yaml with docker compose up -d --build note that if you have one of my previous compose.yaml you need to remove the volumes line from it.

Notes:

• ~As far as I understand, configuration options from config.json and accounts from accounts.json are 'embedded' in the script when it is built. I'm not certain if env vars provided in your compose will still 'overwrite' the default values.~
• ~If anyone can test, that'd be appreciated. Test by making some changes in your compose that do not match your config.json and check the script logs via docker logs netsky - for example add more clusters, that's easy to check on run.~
• ~I'm working on better handling of docker env vars for both the config and accounts~ Done, new push coming shortly
• My next job is to make sure that accounts can also be provided directly in the compose.yaml and 'fed' to the script while it is built. This would mean a user could do everything in compose, and it's all handled automatically. Hopefully.

[mgrimace]

OK @OrangeChu I just pushed a change to my docker test branch that should: 1) let you build the script fully within the container (no local node.js install, etc.) 2) let you specify config options in your compose.yaml that are passed into the script at build time.

Recap of steps to install latest build:

1. Remove /node_modules and /dist from your Microsoft-Rewards-Script folder (which is where the script was 'built' locally)
2. Remove existing containers and images: docker stop netsky && docker rm netsky && docker system prune -a -f
3. Optionally, remove node.js/previous dependencies from your system
4. Source code is still required to build. Clone my repo (which has the updated Load.ts file required for loading docker env vars). Alternatively clone Netsky's repo and just replace the Load.ts in /utils. Don't use Netsky's latest release, AFAIK it doesn't have some other updates that I pushed after the release was created (namely the scheduling stuff for docker).
5. Use my updated compose.yaml with docker compose up -d --build note that if you have one of my previous compose.yaml you need to remove the volumes line from it at minimum.

[mgrimace]

Ok, when I run the script with any docker command (e.g., docker restart netsky) it runs as expected but my cron job did not load my env vars and tried to start with the unmodified config.json options, which included HEADLESS=false, which of course doesn't work.

If anyone is testing, please check your logs and see if the script ran as expected.

I'm not sure if this is a bug or something I did by accident locally (I've been messing with a lot of things).

An alternative could be to update config.json with the options specified in the compose. I can do this using a script and a line in the Dockerfile before the build line.

I'm thinking this would actually be better, as it would update the config for the user. Also, I can revert the changes I made to the Load.ts (so it remains untouched).

I'll push that change to my test branch later today after I double-check some things.

[mgrimace]

Ok folks, it was more complicated than I thought. The short answer is that no, it never required the script to be built locally, but if the script had been built locally at some point, then it was pulled into docker as it was in the working directory and caused the problems above (AFAIK). I added a brief instruction to the readme to remove the appropriate folders before starting the container if switching to Docker.

I reverted much of the changes in my recent push on my test docker. They weren't necessary and made things worse with cron. I was able to build it just fine without a local build of the script on a clean Debian system.

The side-effect is that docker env vars don't actually work as expected. They can over-write the values when there is a full script imported into docker, but the env vars can't be used if the script is built in docker (or at least I can't figure out how). AFAIK, a weird side-effect of building it locally first meant you could use env vars, but if you build it in docker than nope.

So, I removed them from the table on the readme while I go back to the drawing board. Updated compose as well with the values that are actually configurable (run on start, and cron time).

Tl;dr customize config.json and accounts.json as you would normally. Remove any remnants of the old script by removing /node_modules and /dist from your Microsoft-Rewards-Script folder. Use my new compose. docker compose up -d.

Testing continues, and officially I need some help from here, I'm not sure I can move this forward anymore myself.

[mgrimace]

Ok new push coming with better env var handling. I created a small .js script called updateConfig.js that updates /dist/config.json in the CMD of the Dockerfile with any values specified in the compose file. This has been working for me both at script launch and scheduled fixing earlier issues I had introduced. The added benefit is that the container ~does not~ should not need to be rebuilt if you change vars in the compose.

This is ideal, for example, let's say you've been using docker and now you want to test disabling Scroll Random Results. You would just make that change in the compose file, then docker compse up -d to quickly restart the container with the change. No need to add the --build flag or delete the old image, it should speed it up for you considerably.

I'll push to my test repo shortly, then if no major bugs or problems, I'll reopen a PR here.

[mgrimace]

Ok, new better env var handling pushed to my main branch (no longer user docker branch), here: https://github.com/mgrimace/Microsoft-Rewards-Script

You should be able to:

• build and run the script entirely within a docker container
• set any config option that you want as non-default using environmental variables (no need to touch the config.json, but you can do it there instead. values set in your compose will take priority, for example if you quickly want to test an option).
• quickly change config options in the compose.yaml, then docker compose up -d to re-run the script using those changes (no need to delete/rebuild the image)

I'll submit as a PR here after a bit more testing.

update: minor but important update to env var names in your compose.yaml. You no longer need WORKERS_ or SEARCHSETTINGS in front of values that fall under those categories in the config.json. For example, change your compose env var from SEARCH_SETTINGS_SEARCH_DELAY_MIN to SEACH_DELAY_MIN. You'll need to update your compose.yaml for any of these options. All values are now updating properly for me.

[OrangeChu]

Ok, new better env var handling pushed to my main branch (no longer user docker branch), here: https://github.com/mgrimace/Microsoft-Rewards-Script

You should be able to:

• build and run the script entirely within a docker container
• set any config option that you want as non-default using environmental variables (no need to touch the config.json, but you can do it there instead. values set in your compose will take priority, for example if you quickly want to test an option).
• quickly change config options in the compose.yaml, then docker compose up -d to re-run the script using those changes (no need to delete/rebuild the image)

I'll submit as a PR here after a bit more testing.

update: minor but important update to env var names in your compose.yaml. You no longer need WORKERS_ or SEARCHSETTINGS in front of values that fall under those categories in the config.json. For example, change your compose env var from SEARCH_SETTINGS_SEARCH_DELAY_MIN to SEACH_DELAY_MIN. You'll need to update your compose.yaml for any of these options. All values are now updating properly for me.

A suggestion is to mount the configuration by mounting the volume. There are too many environment variables now.

compose.yaml

  volumes:
      - /volume1/docker/microsoft/accounts.json:/usr/src/microsoft-rewards-script/dist/accounts.json
      - /volume1/docker/microsoft/config.json:/usr/src/microsoft-rewards-script/dist/config.json

And, after this configuration, there is no need to use updateConfig.js anymore.

[mgrimace]

Ok, new better env var handling pushed to my main branch (no longer user docker branch), here: https://github.com/mgrimace/Microsoft-Rewards-Script You should be able to:

• build and run the script entirely within a docker container
• set any config option that you want as non-default using environmental variables (no need to touch the config.json, but you can do it there instead. values set in your compose will take priority, for example if you quickly want to test an option).
• quickly change config options in the compose.yaml, then docker compose up -d to re-run the script using those changes (no need to delete/rebuild the image)

I'll submit as a PR here after a bit more testing. update: minor but important update to env var names in your compose.yaml. You no longer need WORKERS_ or SEARCHSETTINGS in front of values that fall under those categories in the config.json. For example, change your compose env var from SEARCH_SETTINGS_SEARCH_DELAY_MIN to SEACH_DELAY_MIN. You'll need to update your compose.yaml for any of these options. All values are now updating properly for me.

A suggestion is to mount the configuration by mounting the volume. There are too many environment variables now.

compose.yaml

  volumes:
      - /volume1/docker/microsoft/accounts.json:/usr/src/microsoft-rewards-script/dist/accounts.json
      - /volume1/docker/microsoft/config.json:/usr/src/microsoft-rewards-script/dist/config.json

And, after this configuration, there is no need to use updateConfig.js anymore.

Yes, that’s a fantastic suggestion for a largely static and portable config.json and accounts.json!

For my use, I’ll keep updateConfig.js because I tend to toggle on/off various features in testing (e.g., DO_MORE_PROMOTIONS=false for #111) and doing so via the compose is the easiest way for me. It also triggers a recreate of the container on docker compose up -d which implements the changes.

It’s a great idea, and do you want to do a quick PR on my repo with a readme change adding your solution? I can write it up something quick if you’d prefer not to.

I’ll mention too that only variables that are changed from default and not already entered manually in the config.json need to be included in the compose. My compose for example only has timezone, run on start, and more promotions for example. The rest are hard-coded in the config because they never change (e.g., headless, min/max delay). Basically, just the ability to make quick temporary changes.

that being said I like the idea of the volume mount and pulling config and accounts out of the code source folder and I’ll be doing that too. Great work

[OrangeChu]

@mgrimace I encountered a strange problem while running tasks on Docker on a scheduled basis. I wonder if you have encountered it. The error is as follows:

Sleeping for 47 minutes (2865 seconds)...
Starting script...

> microsoft-rewards-script@1.4.6 start
> node ./dist/index.js

[5/28/2024, 4:47:51 AM] [PID: 33] [LOG] [MAIN] Bot started with 1 clusters
[5/28/2024, 4:47:51 AM] [PID: 33] [LOG] [MAIN-WORKER] Started tasks for account xxx@xxxxxx
node:internal/process/promises:289
            triggerUncaughtException(err, true /* fromPromise */);
            ^

browserType.launch: Executable doesn't exist at /root/.cache/ms-playwright/chromium-1117/chrome-linux/chrome
╔═════════════════════════════════════════════════════════════════════════╗
║ Looks like Playwright Test or Playwright was just installed or updated. ║
║ Please run the following command to download new browsers:              ║
║                                                                         ║
║     npx playwright install                                              ║
║                                                                         ║
║ <3 Playwright Team                                                      ║
╚═════════════════════════════════════════════════════════════════════════╝
    at Browser.createBrowser (/usr/src/microsoft-rewards-script/dist/browser/Browser.js:20:61)
    at MicrosoftRewardsBot.Desktop (/usr/src/microsoft-rewards-script/dist/index.js:99:51)
    at MicrosoftRewardsBot.runTasks (/usr/src/microsoft-rewards-script/dist/index.js:83:24)
    at MicrosoftRewardsBot.run (/usr/src/microsoft-rewards-script/dist/index.js:51:18)
    at /usr/src/microsoft-rewards-script/dist/index.js:197:9 {
  name: 'Error'
}

Please note that this error only occurs during scheduled runs. If I choose to execute immediately when running Docker, this error will not occur.