Plan for this project

[Akarys42]

Hello Traveler!

This was the original idea back in the days when the project was called server-bootstrap. This is kept for archiving purpose!

The original proposal

Abstract

Here is the plan that I have in mind for this project, it should get updated as we go along. The plan for this project is to have a simple web interface allowing you to (re)sync development servers and generate config files for it.

Server generation

When a new server is generated, a bot will create a new server, gather the current state of PyDis, recreate any channel, roles, emotes and webhooks needed for a working test server. Then, the bot would generate a config file using the new IDs. This config file should be as succinct as possible, and do not override any potentially user defined setting, which is currently impossible (see below). The bot would then add to the guild the user who requested the new server using the OAuth token they provided when requesting the server, send a message with some useful links (contributing page, a link to re-invite the bot to this specific guild, and the song “lint before you push” of course) along with the config file, then leave.

Re-syncing

servers When the bot is invited to a server, either using the link it dropped when generating the server or using an invite link on the website, it will ask users with the Manage Server to confirm the changes, and will proceed to create any missing objects, and delete any non-referenced objects, generate the config again, and send the end message again.

Getting the state of PyDis

This is the current problem. We want to only have as little things on the server as possible, so we thought about using the config file on the bot repo, but the current format doesn’t allow us to have enough information to generate a good test server (lack of the channel order, role colors, emote source file...). Also, because the current setup uses YAML anchors, it is impossible to have a config file with only the stuff provided by discord. Another thing I’d to point out, if we decide to use a file from another repo using the GitHub API or something similar, we would have to allow users to select a remote and a branch to choose from (defaulting to python-discord/master). I think mark has some ideas about refactoring the bot config.

[MarkKoz]

I don't agree with trying to sync with our main guild. The test guild should be generated purely from the config. I believe this will be feasible if the config format/system is changed to accommodate for easy distinction of types for IDs (i.e. channel vs. category vs. webhook vs. role). I proposed some ideas before: using type annotations, a naming scheme (e.g. "role_" prefix), and nesting them under aptly-named keys.

Channel order doesn't matter. Role colours don't matter. Missing emotes don't really matter, but it may be useful to get an authentic preview of the front end for a feature. If it's important enough, we can look into always including the URL for emojis.

Keeping a guild in sync would be a matter of checking if anything is missing when the bot starts. It can then either automatically create missing things or show a log/message that says "run this command to sync things". What you propose sounds more like the user/role sync confirmation that was recently removed. That's fine too, though relatively more complicated to implement. I see no reason for a web interface.

Keep in mind a large portion of channels are only there for whitelist/blacklist purposes, so they aren't strictly necessary. In fact, many if not all features don't need separate channels; they are only separate for organisational purposes. We may be able to cut down on generated channels by re-using some channels for multiple config fields. I don't have any concrete ideas yet, but one that comes to mine is setting a flag somehow that indicates which channels should preferably be separate.

[SebastiaanZ]

Why would we maintain two versions of the "required" state? We already keep a lists of configs for our main server that could serve as a model for a test server. It would mean that you don't reuse channels, but it saves us on having to maintain two constants states and it decouples the state of our test server with the model we use for other test servers. The state of our main guild should always be in sync with the HEAD of "master", while our test server could be the home of experiments for things yet to be merged.

If we were to have a config file that looked something like:

server:
   id: 12345566

   categories:
       information:
           id: 12344235

           channels:
                announcements: 230409432
                news: 349910832970842

and so on for the main guild, you'd have everything you need to set up a test server with the current state of "master" without actually "looking" at the main guild directly. You just generate everything based on the configs we necessarily maintain for required features anyway.

A side-effect is that you could maintain the same category/channel order, but that's by no means required. It does mean that the right channels would be in the right categories, which would make sense, as we're starting to use categories more to whitelist/blacklist things.

[Akarys42]

@MarkKoz

I don't agree with trying to sync with our main guild. The test guild should be generated purely from the config. I believe this will be feasible if the config format/system is changed to accommodate for easy distinction of types for IDs (i.e. channel vs. category vs. webhook vs. role). I proposed some ideas before: using type annotations, a naming scheme (e.g. "role_" prefix), and nesting them under aptly-named keys.

Yeah, this sounds like the best approach so far, by "syncing with the mean guild" I didn't mean to grab the status from Discord directly.

Channel order doesn't matter. Role colours don't matter.

It doesn't matter, but it is still useful to have. I mean, it is still better to feel "at home" in the dev server.

Missing emotes don't really matter, but it may be useful to get an authentic preview of the front end for a feature. If it's important enough, we can look into always including the URL for emojis.

It actually is. The most notable is probably the trash can emoji since it can create quite a lot of exceptions when playing around with the bot. Also, I think they have to be actual emojis because they are used inside embeds, but I might be wrong about that.

Keeping a guild in sync would be a matter of checking if anything is missing when the bot starts. It can then either automatically create missing things or show a log/message that says "run this command to sync things". What you propose sounds more like the user/role sync confirmation that was recently removed. That's fine too, though relatively more complicated to implement. I see no reason for a web interface.

So we would have this feature directly on the dev bot? The idea was to automate things as much as possible, and we need a (simple) web interface to get OAuth tokens, that was @jb3 idea, I quite like it personally.

Keep in mind a large portion of channels are only there for whitelist/blacklist purposes, so they aren't strictly necessary. In fact, many if not all features don't need separate channels; they are only separate for organisational purposes. We may be able to cut down on generated channels by re-using some channels for multiple config fields. I don't have any concrete ideas yet, but one that comes to mine is setting a flag somehow that indicates which channels should preferably be separate.

Sure, we could remove quite a few channels, but while we are at it, I don't see why we wouldn't have every channel.

@SebastiaanZ

Why would we maintain two versions of the "required" state? We already keep a lists of configs for our main server that could serve as a model for a test server. It would mean that you don't reuse channels, but it saves us on having to maintain two constants states and it decouples the state of our test server with the model we use for other test servers.

That was the idea, using what is already here.

The state of our main guild should always be in sync with the HEAD of "master", while our test server could be the home of experiments for things yet to be merged.

That is why I think we should be able to choose if we want to sync based on python-discord/master, or we can specify another remote/branch. Also, let's not forget that while this tool is intended to be used by us in our current workflow, It will mainly be used by external contributors that don't want to maintain a dev server themselves.

I quite like the config idea personally.

[MarkKoz]

Why would we maintain two versions of the "required" state? We already keep a lists of configs for our main server that could serve as a model for a test server. It would mean that you don't reuse channels, but it saves us on having to maintain two constants states and it decouples the state of our test server with the model we use for other test servers.

Sure, we could remove quite a few channels, but while we are at it, I don't see why we wouldn't have every channel.

Because having channels that aren't strictly necessary just adds clutter and makes the test guild more cumbersome to use. As an example, consider the channels recently added for blacklisting duckpond. One only needs 1-2 channels to test if the blacklist works, and those channels can be existing channels used for other features.

So we would have this feature directly on the dev bot? The idea was to automate things as much as possible, and we need a (simple) web interface to get OAuth tokens, that was jb3 idea, I quite like it personally.

Seems like a lot of complexity just to save someone having to create a blank server and invite the bot. It could be a standalone CLI app, but I think it's also a good idea to have the bot be able to detect something out of sync too. Otherwise, people will forget to sync.

It doesn't matter, but it is still useful to have. I mean, it is still better to feel "at home" in the dev server.

If we were to have a config file that looked something like:

Unfortunately, trying to do this doesn't fall in line with my idea of a config overhaul, which was basically the following:

Define a config class in any module. A decorator or superclass will register it as a config class in a global table. In this case, the names' prefixes are used. It could be done with type annotations instead.

@config
class Config:
    maximum_thing = 1
    channel_general = 12345656778
    voice_general = 1223445423
    category_help = 2342342348
    role_admin = 2343234

Once registered, read from the config file and see if there are overwritten values that need to be assigned. A config file would look like this.

guild:
    id: 12312313
    categories:
        help: 2342342348
    text_channels:
        general: 12345656778
    voice_channels:
        general: 1223445423
    roles:
        admin: 2343234

bot.exts.my_ext:
    maximum_thing: 1

Note that a config file wouldn't be necessary if one wishes to use purely the default values as defined in the config classes. There's no order/category information for the channels because it would be cumbersome to require that to be specified for every channel defined in a class.

[Akarys42]

@MarkKoz so you’re saying that it should just be a cog on the development bot?

[MarkKoz]

I think that should be one component of it. The cog can just be used to check if the guild is out of sync and resync it.

However, this project can do more than sync the server. It can have an interactive setup prompt that automatically sets the right config values based on questions like "are you using docker compose?". In this regard, a CLI app has the advantage of being simpler to implement and the ability to directly modify local files. A website would have a nicer UI, enabling us to show screenshots of how to fork, create a bot app, etc. as well as get OAuth2. Such website could technically run locally too but this sounds error prone and is relatively heavy on dependencies.

[Akarys42]

I think that should be one component of it. The cog can just be used to check if the guild is out of sync and resync it.

I feel like keeping everything in this project is a better idea, or we would have a lot of duplicate code between this project and the bot.

However, this project can do more than sync the server. It can have an interactive setup prompt that automatically sets the right config values based on questions like "are you using docker compose?". In this regard, a CLI app has the advantage of being simpler to implement and the ability to directly modify local files. A website would have a nicer UI, enabling us to show screenshots of how to fork, create a bot app, etc. as well as get OAuth2.

I was thinking about having an API so we can have bootstrap scripts like described in python-discord/organisation#221, but we will have to see about that later.

@MarkKoz would it be possible for you to draft an implementation of this new config format soon-ish?

[MarkKoz]

I feel like keeping everything in this project is a better idea, or we would have a lot of duplicate code between this project and the bot.

I was imagining this to be a dependency, so the cog could be imported from this package.

would it be possible for you to draft an implementation of this new config format soon-ish?

Yes

[Akarys42]

I was imagining this to be a dependency, so the cog could be imported from this package.

I don’t think it is a good idea for two reasons, it adds unneeded complexity for both production and development, and if the config format changed and the bot can’t boot anymore, you’re pretty much screwed. But I’m not really strongly opposed to that.

[MarkKoz]

I wrote up a demo here.

It works based off attribute names. This isn't the final decision, but it felt the easiest to implement for the sake of the demo. That being said, I do feel we're going to end up naming attributes with such scheme anyway just for the sake of readability, so why not leverage that further? For example, Config.help would be vague, but Config.help_channel is clear. Could also go with Config.Channels.help (similar to the system in production currently) but that seems more cumbersome to define.

As things currently stand with the demo, it's nice that there are no imports required to define the config; there are no custom types for annotations or any config base classes. Something else to note is that it's still possible to have e.g. a channel constant that is put under the extension's key - just don't use the _channel suffix. Unfortunately, this will not work the other way around - using the suffix for something that isn't a channel. I don't foresee it being hard to come up with alternate names to work around that.

The config class is in the extension testbot.extensions.sample:

class Config:
    my_channel: int = 400121469641818124
    my_category: int = 400121469641818123
    my_role: int = 400499713570242571
    my_voice: int = 647317854978310157
    my_webhook: int = 690000402552455210

    enabled: bool = True
    role_whitelist: List[int] = [400499713570242571, 400121903420801033]

Assuming the user wanted to customise all values, the config file would be

guild:
  categories:
    my: 11111
  text_channels:
    my: 2222
  voice_channels:
    my: 3333
  webhooks:
    my: 4444
  roles:
    my: 5555

testbot.extensions.sample:
  enabled: false
  role_whitelist: [111, 222]

but it can be shorter if default values are desired for some of the attributes.

It's possible to change it to keep channels, roles, etc. under each extension but I thought it was nicer to group those together. One thing that needs to be worked out is loading the config for extensions which are not loaded by default, but may be loaded via a command later. I have some simple ideas for that but didn't get around to it tonight. I also haven't worked out anything for a global config yet.

[MarkKoz]

it adds unneeded complexity for both production and development,

Developing and deploying a website sounds just as complicated, if not more, than a PyPI package.

and if the config format changed and the bot can’t boot anymore, you’re pretty much screwed

I'm not sure what you mean. Keep in mind, with my proposal, everything would have a default value without any YAML config needed. Existing configs would only break if an attribute was renamed or moved (or we completely overhauled the format again, which is unlikely), but this seems unavoidable.

[Akarys42]

The idea of a PyPi package sounds good.

The config looks really good too, there's just one thing that should work differently in my opinion ; in your example, you have duplicates for the role_whitelist parameter, if we do have duplicates for every YAML anchors we have right now, this would turn into hell pretty quickly, and the generated config file would have to redeclare all those attributes, it would be nice to have a way to re-use a parameter, using a string or so.

[MarkKoz]

That's a great point. While it can easily be fixed by extending the default list, how would one distinguish between wanting to extend and wanting to overwrite?

[Akarys42]

Is there a case where we would want to extend a list? I feel like if we do specify it for some reason, we would like to override it.

[MarkKoz]

Sorry, I must have misunderstood the point of your original comment. Can you rephrase it?

[Akarys42]

Alright, so after some discussions in #dev-core, here is the current plan:

• This project will host a PyPi package for every bits and pieces shared across the bots that use this config system
  • Although, should we make the code loading and updating the configs inside this package? I'd say yes
• We put the website as optional, and we will probably work on it after the first release, we'll see