search

Chaosthebot / Chaos

A social coding experiment that updates its own code democratically.
http://chaosthebot.com
MIT License
2.44k stars 210 forks source link

Chaosbot V1 experiment conclusions and future direction #553

Open amoffat opened 7 years ago

amoffat commented 7 years ago

The chaosbot v1 experiment has run its course. The bot is currently dead. There was a very strong force of activity from launch until very recently, and I would like to outline what I believe are the weaknesses in v1 that led to this point:

  1. Lack of a complete development environment that mirrored production. This one was crucial for ensuring that changes to be made were deployed smoothly.
  2. Lack of tests from the start. This would have resulted in fewer breaking bugs being merged in.
  3. Lack of a standardized way of committing private data. For example, reddit or twitter credentials. This only came later, and had some hiccups.
  4. Lack of segregation between critical and non-critical code. It was very easy to break the system because critical code was alongside non-critical code.

Where to go from here

This is up to the community. If the community is interested in a version 2 that tackles those previous 4 points, I think we should take a step back and address them at a slower pace. This could involve including contributors on the chaosbot project, to directly commit to master. The idea would be that we build towards a v2, then when it has the basics required to self-sustain, we release it into the wild again.

Thoughts, criticisms, critiques? Everything is valid, this was a project driven by us all, after all :)

PlasmaPower commented 7 years ago

Lack of segregation between critical and non-critical code. It was very easy to break the system because critical code was alongside non-critical code.

This is what I'd really want for a v2. Ideally, it'd even be split into separate repositories. We could then do stuff like have different voting thresholds for different repos.

dbpokorny commented 7 years ago

@amoffat @PlasmaPower may I suggest some Kierkegaard?

Specifically Fear and Trembling / Repetition.

rudehn commented 7 years ago

Sad to hear. This was a lot of fun!

I think part of the problem was that there was a lot of activity when simple updates were required (like updating a readme), however when more heavylifting needed to be done (such as the database), not as many people felt like they could contribute. Furthermore items like the database created a bottleneck in development; there were PRs that had to wait until the db had been implemented before their completion could be finished.

dbpokorny commented 7 years ago

@amoffat @PlasmaPower @rudehn bot running on chaosbot.org

Actually, it is trying to run, crashing a lot

I backed out the "reset git history if I crash" behavior because it makes testing difficult

Edit: repo is dbpokorny/chaos

dbpokorny commented 7 years ago

update: I don't know how to run supervisor or remove the code that depends on supervisor, so until that happens, I'll have to restart the bot manually every time it merges a PR

amoffat commented 7 years ago

@rudehn good points. A database with an easy way to do migrations seems to be another pain point to address

dbpokorny commented 7 years ago

@amoffat what is the motivation for using supervisor in the first place?

amoffat commented 7 years ago

@dbpokorny Open a new issue for any off-topic questions. I want to keep this conversation on-topic

dbpokorny commented 7 years ago

/vote close

mdcfe commented 7 years ago

Would v2 be based on v1, implementing the above points, or would it be an entirely new codebase?

Also, I think v2 should require full documentation of APIs and proper error handling in PRs to prevent the crashes and confusion which occurred in v1.

PlasmaPower commented 7 years ago

I think it should be a rewrite. If we implement the above points, it'd pretty much have to be.

I don't think full documentation is necessary for APIs this shallow. Most of our API functions, especially the GitHub ones, are only a few lines long. It's easier just to read the code.

PlasmaPower commented 7 years ago

Also, should we keep v2 in Python? Switching to Node.JS might attract new contributors. JS, like Python, is widely known, which is why I'd recommend Node.JS if we decide to switch.

mdcfe commented 7 years ago

@PlasmaPower I think Node would absolutely attract new contributors, but we might want to use more robust languages for critical code.

Regarding documentation, I was referring to things such as the apparently-confusing encryption module and the database.

Also, I support the idea of independent wrappers and possibly also containers. For example, a repo for the component responsible for managing containers and a separate repo for the web server component, and maybe even one for GitHub events which feeds back to other components via a message queue.

mark-i-m commented 7 years ago

Rust. :+1:

phil-r commented 7 years ago

Lack of tests from the start. This would have resulted in fewer breaking bugs being merged in.

That what stopped me from contributing more. New features were added without any tests for them(or even actually trying to run the code). I've tried to add tests to some critical parts, but it's quite boring to just write tests.

mdcfe commented 7 years ago

Just going to summarise suggestions so far (please correct me if I'm wrong or miss anything):

Brief summary

Last updated on 18th June 2017.

Flaws with v1

amoffat

rudehn

md678685

phil-r

mark-i-m

anythingbot

Suggestions for v2

☑️ = confirmed ❓ = unlikely

amoffat

PlasmaPower

md678685

mark-i-m

Redmega

Swizz

andrewda

anythingbot

Changes

20170618

https://github.com/chaosbot/Chaos/issues/553#issuecomment-308453435
https://github.com/chaosbot/Chaos/issues/553#issuecomment-308597268
https://github.com/chaosbot/Chaos/issues/553#issuecomment-308654484
https://github.com/chaosbot/Chaos/issues/553#issuecomment-308753168
https://github.com/chaosbot/Chaos/issues/553#issuecomment-308959717
https://github.com/chaosbot/Chaos/issues/553#issuecomment-309200370
https://github.com/chaosbot/Chaos/issues/553#issuecomment-309201806
https://github.com/chaosbot/Chaos/issues/553#issuecomment-309201995
https://github.com/chaosbot/Chaos/issues/553#issuecomment-309224235
https://github.com/chaosbot/Chaos/issues/553#issuecomment-309226401
https://github.com/chaosbot/Chaos/issues/553#issuecomment-309232338

20170618-1

https://github.com/chaosbot/Chaos/issues/553#issuecomment-309301764

20170618-2

20170618-3

https://github.com/chaosbot/Chaos/issues/553#issuecomment-309303103
Redmega commented 7 years ago

Sad to see that the experiment ended so soon before I had a chance to add something to the site, I was looking forward to incorporating React and doing some cool stuff (Had planned for a twitter feed on the homepage). Once v2 is launched with the MVP features I'll try again. 😃

We should include some sort of code coverage tool and only merge if code coverage stays above a certain percentage. (95%?)

I second the vote to incorporate Node. I use Node.js in work and personal projects and it's definitely a popular and easy to start language. Its easier to control code style (ala prettier, eslint, etc), the syntax is familiar to many, and writing idiomatic javascript is easier to grok imo, whether you're writing functional, reactive, imperative, or what-have-you. I had trouble writing idiomatic python in my playground scripts so I didn't want to commit to the python part of Chaos (A personal problem, I know, but still, it might be something that others also felt).

As for multiple languages, I think the easier way to do it would be to have Node.js do everything, but I think a lot of the people who contributed to the core chaos functionality would rather see it stay in Python. I think a combination python+node would be good. Rust is unknown to me but it would be a learning opportunity for sure. I'm down for that as well.

mark-i-m commented 7 years ago

I think there is one other thing that was lacking (which may have been part of the experiment): There was no way to actually define a purpose for chaosbot. @anythingbot's description here was surprisingly accurate. It would be nice if there was some mechanism to force people to actually implement something, rather than just a self-perpetuating system (which is cool, but not particularly useful).

amoffat commented 7 years ago

That's an interesting idea @mark-i-m. Maybe some kind of simple project management system where deciding/creating new sprints requires a vote

Swizz commented 7 years ago

+1 for the multi repo.

We tried to create one experiment, but I guess, it can be many experiment : the core, the democracy, the website, etc...

I personally focused on the developer experience like enhanced github feedbacks, etc... Some, focused on the democracy system, others only on the website.

Each purpose can be result to an auto managed team, self board, self democracy system, etc...

Redmega commented 7 years ago

If it's multi repo I think the switch to make chaosbot an org would make that easier, right?

Swizz commented 7 years ago

But.. to be honest, I think the Chaos hype is over, isnt it ?

Redmega commented 7 years ago

I'm still excited for it. I'm sure with a v2 release we'll get a new influx of contributors.

mdcfe commented 7 years ago

I'm sure with a v2 release we'll get a new influx of contributors.

Seconded, especially if we switch to Node.js and get new contributors onboard.

Swizz commented 7 years ago

@Redmega @md678685 If we move to multi repo, we may develop multiple core in multiple langage and let the community vote for switching from one to another ? o/

mdcfe commented 7 years ago

@Swizz I think the repos could start empty and we could open PRs to it in each language implementing the same points, and do a manual democratic vote for the core to accept?

Redmega commented 7 years ago

@md678685 @Swizz I feel like that might be too much inventing of the wheel, though. And with different cores in different languages we'll quickly see a difference in actual functionality as some languages implement some features, and others implement different ones. To some this might be a good idea, but I think the actual core which runs the self-updating mechanism should be one single codebase, whatever the language.

Modules can be added to run a webserver, an api endpoint, or what-have-you, and those can be swapped around, abandoned, or whatever, but I strongly believe the core functionality shouldn't follow the same methodology.

That being said, if we could define what exactly the core functionality is, we could add modules to THAT in whatever languages or frameworks the community votes for (i.e. if the core functionality is just approving pull requests, we could have voting modules with different voting rules, messaging modules with different messages, more emojis, etc., and build upon it from there).

mdcfe commented 7 years ago

I think one of the major things to do is deciding exactly what the "core" and "modular" functionality should be. My suggestions:

Core

Modules (one repo each)

amoffat commented 7 years ago

So which option sounds most appealing to people?

a) Revert chaosbot to the the most recent working state, then work towards v2 milestones using the PR democratic process. (Vote on this comment with :+1: )

b) Convert chaosbot to an organization and add a select team of core contributors (probably based on the meritocracy list) with write commit access. We can then work towards v2 more directly. (Vote on this comment with :heart: )

Swizz commented 7 years ago

If I choose b, but I am not a Meritocrate yet, how can I contribute ? With PR ? Meritocrates will be the ones who can validate or not my PR ?

Pretty frustrating, no ?

Couldnt we start a brand new code base for V2 using V1 for democracy ?

Redmega commented 7 years ago

The issue with doing it democratically is it leaves the possibility open that people will come and add changes which don't further v2, or worse, which hinders v2 progress. We'd have to stay vigilant in our voting.

I wouldn't mind contributing via PR.

I'd vote :heart: but I'm on mobile

mdcfe commented 7 years ago

If v2 is in a separate branch, then v1 could be used for the democratic process. However, I would prefer that it is multi-repo and in an organisation. We could give the most recent meritocracy write access, and then they can merge PRs from other people as requested.

anythingbot commented 7 years ago

@amoffat why don't you just ban anyone not on the meritocracy from contributing to chaosbot/chaos? What do you get with an organization that you can't get from controlling the set of people who are allowed to have a vote and allowed to comment on issues and PRs in chaosbot/chaos?

andrewda commented 7 years ago

I agree with the organization, but I think meritocrats should be encouraged to open PR's for all changes they wish to make instead of committing directly to the repository, unless it's a hotfix. Just to make sure everything is done (mostly) democratically.

anythingbot commented 7 years ago

@amoffat I would add to your list the following

  1. A lack of transparency in the moderation of github PR comments and issue comments. Currently there is no way to get information about the moderation settings for either, including restrictions or bans on specific users, unless @amoffat makes his moderation decisions public. If the chaosbot were moderating the community itself, we could instruct it to report all moderation decisions.

  2. A lack of due process in the moderation of github PR comments and issue comments. Transparency alone merely informs the community of decisions made by moderators. Due process means moderators cannot act to restrict or ban users without going through a bureaucratic process designed to limit the power of moderators. In theory, the chaosbot code can guide moderators and hold moderators to a standard set by the community, but it means moderators will have to interact with the chaosbot directly (presumably through chaosthebot.com) and let the chaosbot communicate with github to enact the moderation decisions.

mdcfe commented 7 years ago

@anythingbot I don't think there is any moderation of the issue and PR comments. I don't think there should be, either - the project is supposed to be democratic, and I think moderation would hinder that.

@andrewda But we'd need a bot to manage the democratic PR merging process in the meantime - would we have to retrofit v1 to support multiple repos until v2 is ready?

andrewda commented 7 years ago

@md678685 Sorry, yea you're right. I was thinking more long-term, once v2 is working.

dbpokorny commented 7 years ago

@md678685 Blocking a user from your personal account is possible (the chaosbot/chaos repository is a personal github account of @amoffat held under the name @chaosbot ).

mdcfe commented 7 years ago

@dbpokorny That doesn't mean that it is actually done, nor does it mean that it should be done.

dbpokorny commented 7 years ago

@md678685 Does contemplating action count as moderation? If a moderator reads posts and considers taking action but doesn't, does this count as "moderation" or "moderation activity"? If you are doing mental work in the context of your role as moderator, then are you moderating? It is hard work deciding how and when to apply controls. I can see why you might think that merely reading comments and contemplating the moderation powers github asks @amoffat to use doesn't necessarily count as moderation. This doesn't change the fact that github places the burden of moderation on @amoffat and expects him to act accordingly when he interacts with projects under his control.

The work of moderation can be organized by the bot, but what happens when we disagree over some aspect of the way we tell the bot to organize moderation? It all depends on whether or not we accept the authority of the bot to moderate a discussion of changing the way the bot moderates, using its current moderation policies. If there is a change in moderation policy, but the nature of the change appears, from the point of view of the bot, to be inappropriate behavior, then the bot will apply moderation controls, potentially affecting the outcome of a decision that affects its own behavior. This is the problem the bot faces: moderating the discussion over changes to its own moderation policy.

Even if @amoffat doesn't act to apply moderation controls, we all expect him to be prepared and to be able to do so when he judges the situation warrants action. We're still putting him under a lot of pressure, and his moderation decisions will affect his reputation.

And, so the thinking goes, he is obligated to attend to progress on the project and remove obstacles when they appear and block progress. He isn't doing this just for himself, he's doing this for us as well; he is creating the space for us, and when we comment in his personal repositories, we are his guests.

dbpokorny commented 7 years ago

@md678685 the problem is roughly that outlined in the film 2001: A Space Odyssey

@md678685 : open the pod bay doors, please ( @amoffat or @chaosbot )

@md678685 : open the pod bay doors, please ( @amoffat or @chaosbot )

@md678685 : hello ( @amoffat or @chaosbot ), do you read me?

@md678685 : hello ( @amoffat or @chaosbot ), do you read me?

@md678685 : do you read me, ( @amoffat or @chaosbot )?

( @amoffat or @chaosbot ) : affirmative, @md678685

( @amoffat or @chaosbot ) : I read you

@md678685 : accept this pull request, ( @amoffat or @chaosbot )

( @amoffat or @chaosbot ) : I'm sorry, @md678685, I'm afraid I can't do that

mdcfe commented 7 years ago

@dbpokorny I don't see how that outlined the problem at all, and it's not relevant to the discussion.

dbpokorny commented 7 years ago

@md678685 what is your problem? just because you can't speak the language doesn't give you a right to come in here and insult me and tell me my point isn't relevant. Get educated.

mdcfe commented 7 years ago

I have now updated the summary with the new comments.

amoffat commented 7 years ago

Thanks for staying on point @md678685 :+1: Later today I'll be converting the user to an org and adding contributors so we can start knocking off items on the outline this week

mdcfe commented 7 years ago

Late proposal, but can I also suggest that we define a standard config format for launching modules that the core accepts? That way, it would be easier to create new modules that the core can manage.

Edit: updated summary again, now with confirmation emoji

mark-i-m commented 7 years ago

I don't know if this is the right place, but another minor suggestion: the issue commands should be invoked by mentioning @chaosbot:

@chaosbot vote blah

Swizz commented 7 years ago

@md678685 We just need a standard API endpoint ? start(), stop(), restart()

o3LL commented 7 years ago

I'm happy we are switching to nodejs, i was not very comfortable with python!

Swizz commented 7 years ago

Are we ?

I think, we can make a repo without content only for issue/discussion purpose, like it does by Waffle for example.