Getting a daily digest from a Ghost blog?

[Wobak]

Hello,

I'm probably a bit stupid, but I can't get goeland to send emails on a regular basis when new entries appear in the rss feed.

Here is the config part that I used : Docker compose :

  goeland-2024:
    image: slurdge/goeland
    container_name: rss-to-email-2024
    volumes:
      - ./goeland-data/config.toml:/data/config.toml
    networks:
      - ghost

Config.toml relevant parts :

loglevel = "debug"
dry-run = false
run-at-startup = true
[email]
host = "<mailhost>"
port = 587
encryption = "tls"
authentication = "plain"
username = "no-reply@<mydomain>"
password = "<mypassword>"
include-title = true
[sources]
[sources.myblog]
url = "<rssfeed>"
type = "feed"
filter = [ "unseen" ]
[pipes]
[pipes.myblog]
source = "myblog"
destination = "email"
email_to = [ "<myemail>" ]
email_from = "Blog 2024 <no-reply@<mydomain>>"
email_title = "Blog 2024: {{.EntryTitle}}"

I created a first blog entry, and then restarted the container, and I got the email. Great news ! But now, I added 2 more articles and no email was sent (after 48 hours). I restarted the container, got 3 emails. Restarted it again, got 3 emails again. I don't get why no goeland.db is created or why if I keep the container running, no email is sent.

Can you help me understand what's going on?

Thanks,

[slurdge]

Yes,

I think it comes from 2 things:

1. You need to mount the goeland.db. Right now you are mounting only the config. I think by default it is mounted to /goeland.db inside the container, or you can specify the path directly if you want to have more control.
2. You need a cron job in your pipe if you want it to execute regularly. RSS is pull, and I didn't want to hammer servers by default. Add `cron = "" and it should execute (e.g. every hour). If you need a cron schedule, you can use cron guru for example.

I think with these two changes, it should work as you expect.

If you are inclined, I would welcome a full example so I will put it into the documentation for docker. Thanks !

[Wobak]

Thank you for your prompt answer. What format should the goeland.db be created? Is there a command I could use to generate it (I would have expected the container to create it, and then I could copy it and mount it, but when I do a find / -name goeland.db in the container, nothing shows up, so I'm assuming it's not created at all.

Do you have documentation around the cron= line ? I want to understand how much control I have on that ?

Also, can I manually insert post entries in the goeland.db ? I've already spammed my contacts more than necessary with the existing 4 first posts (I guess I could change the email_to for the goeland.db to be populated, but I'd like to have a bit more control :))

Happy to provide the end config & setup once everything works fine if you want to !

[slurdge]

I would have expected the container to create it

I would too. I think this is a weakness, it would be better IMHO to either mount a folder and then instruct goeland to put the db inside it (no need to copy from container), or better yet, goeland should work with 0 byte initial db (that would be created by the bind I guess). As for your immediate problem, try to put database = <pathtoamountedfolder/goeland.db> and see if it gets created/correctly mounted. If not, i would have to try it on my side.

Do you have documentation around the cron= line ? I want to understand how much control I have on that ?

Basically any cron syntax, e.g.:

cron = "30 12 * * *" #execute the schedule every day at 12h30
cron = "*/5 * * * *" ##xecute the schedule every 5 seconds

Also, can I manually insert post entries in the goeland.db

No, and it would be difficult. However you can "populate" it by making the email_to empty and it should mark the entries as seen. I have too "spammed" by friends :-). The tricky part is that you would need to know the key that is used for the db. I guess I could either add:

1. A "see" filter that would mark all entries as seen, empty the list (hence no emails sent). Easy enough to implement, but it makes a back & forth by changing the config file, which I don't like.
2. A "see" command that would do the same thing by flagging entries inside the unsee filter. This is also doable, and would be more consistent with the "purge" command, however this is more work, and usually is not that common enough, so I may be lazy around it. We'll see :-)

[Wobak]

So I tried :

  goeland-2024:
    image: slurdge/goeland
    container_name: rss-to-email-2024
    volumes:
      - ./goeland-db:/data/db/
      - ./goeland-data/config.toml:/data/config.toml
    networks:
      - ghost

And then :

database = "/data/db/goeland.db"

in the config.toml.

Nothing appeared in the folder, so I did a touch of goeland.db in that folder, with a chmod 777, and size remained to 0 after restarting the container.

Not sure what would be the next step. Is there a log I could look at (nothing relevant in docker compose logs despite being in debug mode)?

[slurdge]

Ok I tried to recreate it on my side. I have the following config:

loglevel = "debug"
dry-run = false
run-at-startup = true
database = "/data/goeland.db"
[sources]
[sources.hackernews]
url = "https://hnrss.org/newest"
type = "feed"
# See doc for available filters
filters = ["all", "unseen"]
[pipes]
[pipes.hackernews]
source = "hackernews"
destination = "none"
cron = "*/1 * * * *"

And I run with the following command line (note the '\' on windows): docker run -v .\data:/data -v .\config.docker.toml:/data/config.toml --rm slurdge/goeland It seems to work as expected. So I'm not sure what's different in your scenario. It may be that goeland doesn't autocreate folders, but the bind mount should do so ?

[Wobak]

Hmmm could it be due to permissions due to windows settings ?

I've put UID 1000 as the owner of the config.toml and the data folders, and still nothing happens when it comes to database creation.

Isn't there a goeland command I could use in the container to generate it ?

Current docker compose :

  goeland-2024:
    image: slurdge/goeland
    container_name: rss-to-email-2024
    volumes:
      - ./goeland-data:/data/
      - ./goeland.config.toml:/data/config.toml
    networks:
      - ghost

config.toml:

loglevel = "debug"
dry-run = false
run-at-startup = true

database = "/data/goeland.db"
include-title = true

[sources]

[sources.myblog]
url = "https://<rssurl>"
type = "feed"
filter = [ "all", "unseen" ]

[pipes]

[pipes.myblog]
source = "myblog"
destination = "none"
cron = "* * * * *"

Folders:

-rw-r--r--. 1 1000 1000 2476 Jun 29 23:28 goeland.config.toml
drwxr-xr-x. 2 1000 1000   63 Jun 29 23:27 goeland-data/

[slurdge]

You are right this is a right problem. If you run with debug flags, you'll see (without changing owner): time="2024-06-30T09:07:34Z" level=error msg="cannot open database: error opening seen database /data/goeland.db: open /data/goeland.db: permission denied"

That being said, if I do the following:

cd /tmp
mkdir goeland
cd goeland/
mkdir data
chown 1000:1000 data
nano config.docker.toml #put the same content as above
docker run -v ./data:/data -v ./config.docker.toml:/data/config.toml --rm slurdge/goeland
ls data/

The database is created. I think you are almost there ! I now realize documentation for docker is severely lacking... I would suggest to try exactly as my steps and then try to convert it to docker compose, sometimes docker compose is doing strange stuff behind the scenes.

[Wobak]

Hello,

Unfortunately, behaves the same with a docker run, and no database created. I tried moving to a different folder, doing a new Dockerfile with adding a VOLUME [ "/data" ] into it, using a docker compose volume instead of docker local folder and nothing worked. I'm going to try and run it from a different server to see if I can find another difference.

Also I don't have the error message even with the debug log. Are you running the 0.18.3 version or are you running a dev branch in your example?

(Sinon je suis français aussi, je sais pas si tu veux qu'on essaye de debug ça par un autre moyen que des échanges sur github même si on pourrait y mettre le résultat final).

[slurdge]

docker run --rm slurdge/goeland version
Build Date:
Git Commit:
Version: v0.18.3
Go Version: go1.21.5
OS / Arch: linux/amd64

However - I almost only use debian.

FR: Oui on peut tenter un autre canal, plutot vocal/partage d'écran ?

[slurdge]

My guess if you don't have any message is that even the config file isn't the correct one...

[Wobak]

Same version here :

root @ t340-2: /opt/docker/rss2email 15 # docker exec goeland-2024 /goeland version
Build Date:
Git Commit:
Version: v0.18.3
Go Version: go1.21.5
OS / Arch: linux/amd64

FR: discord? J'ai le même @ que sur github si tu veux m'ajouter. Sinon je te laisse proposer un autre moyen :)

[Wobak]

Ok so the issue lied in the config.toml file. I re-generated one from scratch, changed to match my blog URL and email settings and now it works fine. Thanks for all the help !