Allow override of 'sites/'

[tmtmtmtm]

Is your feature request related to a problem? Please describe.

Since #180 it has been possible to override an individual filename, but as far as I can see there is currently no way to override the directory that files get created within: this always defaults to sites/

Describe the solution you'd like

The simplest approach would probably be for a top-level config option to change this for all sites, though it might also make sense to be able to override this for an individual site with a config option at that level.

[EdJoPaTo]

The main filename (website-stalker.yaml) is also not changeable currently. It has benefits to the complexity with the git handling and so on so I like that its a bit more static.

What would be your use case of having a different folder name there?

[EdJoPaTo]

I also discussed it with @Teufelchen1 so maybe there are some different thoughts?

[Teufelchen1]

Keeping it static and less complex is also in my favor. I also don't see the use case? An elaboration of the intended workflow where a different folder name is useful would be great!

[tmtmtmtm]

@EdJoPaTo @Teufelchen1 thanks for considering this.

I already have my own hand-rolled version of this, where I scrape hundreds of government and parliament sites worldwide, extract lists of people from them, and then compare that against information in Wikidata. Currently each source is scoped to its own directory, which keeps things cleaner and more manageable, and makes it easier to build reporting etc.

I would like to replace large chunks of this workflow with website-stalker, as you have a lot of very nice functionality that my much simpler curl-and-commit version doesn't have, but I would like to maintain the one-directory-per-source structure as much as possible.

[Teufelchen1]

We discussed your idea and concluded it would be a nice to sort the output in folders per domain. Thank you for the additional description! E.g.:

├── sites
│   └── de-edjopato
│       ├── post.html
│       └── robots.txt

Would that work for you?

However, this would be a breaking change, so care must be taken (git history?). We would also like to keep it as simple as possible; If the folder structure shown (one folder per TLD) is non configurable, would it still fulfill your feature request?

[EdJoPaTo]

The downside are many pages on different domains but I think most people have not that many different domains? My public example would have 21 folders which seems fine.

Git handles renames kinda well so I don't think the breaking change would be that annoying.

what's interesting are domains without a path. https://edjopato.de/ for example. Maybe use index for that? with a editor it could also know the extension (html in this case) de-edjopato/index.html.

Also @Teufelchen1 I would remove the sites folder and put everything plain into the main repo. Otherwise there are only .github, README.md, sites and website-stalker.yaml which isn't that much. having it with less directory depth seems better to me?

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.