okTurtles / group-income

A decentralized and private (end-to-end encrypted) financial safety net for you and your friends.
https://groupincome.org
GNU Affero General Public License v3.0
329 stars 43 forks source link

Project feedback #1006

Closed thesoftwarephilosopher closed 3 years ago

thesoftwarephilosopher commented 3 years ago

Hi team. I have some constructive feedback that I've come up with based on my experiences working on this project.

Simplify project features

A good MVP has minimal features, focuses on the essentials, and puts off extras until later. Essentials for this project would be things like proper user accounts, real email-verification, implementing data persistence, and the ability to actually send payments from the app, which it currently has none of. Extras would be things like real-time updates in clients and the chat feature. These should have waited until after public release.

Remove SBP

SBP is just a a much more complicated version of global function calls, with a little bit of RCP mixed in. Using regular ES modules would be far simpler, and make the code much cleaner and easier to work with. It would also have better automatic integration with IDEs. The RCP and logging concepts can be done in other ways that are orthogonal to ES modules and don't introduce this serious complexity, and aren't invasive in the implementation of the code.

And it's not actually an advantage to not have to import functions in order to call them. That leads to implicit dependencies, and it breaks IDE support, where you can Ctrl-Click to go to a function definition. This is extremely useful. And modern IDEs have support to automatically update import paths when you move/rename a function or file.

Decouple logic from Vue

It's generally considered very bad practice to couple business logic so tightly with an external framework. This makes it very difficult to refactor or reuse code, to test business logic, or to structure code properly. I strongly recommend pulling all the business logic out into a group of pure JS functions with its own separate structure, and then call this business-logic pure-JS app from Vue, without ever doing any real logic in Vue that's not puely related to browsers. The rule of thumb is, make sure your business logic can be called from a CLI utility, curl, or a web browser, without ever repeating any code.

Refocus tests

Relating to the last one, Cypress tests should only be smoke tests for a few hot spots in the UI, just to make sure everything is hooked up right. The vast majority of the tests should be written in pure JS for the pure JS business logic. Knowing whether a payment was considered to be partial or not should not be something that requires running the actual UI to figure out.

Simplify configuration

Simple things like supporting project-relative import paths make it hard to integrate with IDEs and add up to a non-trivial amount of configuration. The gruntfile is about 500 lines of code. The general rule of thumb is, the more you customize something, the more you break everyone else's efforts to integrate with it, and the more you glue things together, the less of other people's glue you can use. And generally, their glue and integrations are generally better quality and better tested than any in-house solutions.

I've seen many small and big companies follow this pattern over the past 10-15 years and gradually move more and more to third party solutions, decreasing their in-house integrations. For an app this size, there could be almost no configuration necessary. Besides, your configurations are causing more issues, such as the one where build times increase gradually while developing.

I admit I tend on the side of zero-configuration (see this React example), which is extreme, and I'm not necessarily suggesting that in this app's case, since it might not work out that great with Vue (but it might) and TypeScript (though JSDocs helps a lot). But I'm positive at least a good amount of configuration could be simplified or even removed here, relative to my other suggestions.

Browser devtools and debuggers are extremely useful, and configuration that works against these or even completely breaks integration with these should be avoided.

Focus on the app

Group Income started out as an idea for an app, and at some point the idea of a protocol creeped in, but it makes no sense in this context. Most people do not want a protocol for everything, they want an app. Forgetting about the protocol will allow simplifying a lot of the project goals and also a lot of the project implementation.

Simplify project goals

This project has long-term goals of becoming an entire software platform and even OS for subsets of society, hence SBP and the chat feature, and that's just not a good reason to add these features and implementation complexities into the app this early, while it's not even released.

Use simpler decentralization

Just use a regular database and allow people to host their own instance of the server. One reason I was told this was avoided was because it would also need to be secured, but the amount of effort that went into creating the current decentralization setup far surpasses what would have been needed to just make sure an open source NoSQL database is securely used by the server.

Use TypeScript

It has basically won. It's only a matter of time until you switch over to it, might as well do it now while the Flow setup is already completely broken. Having proper types is really essential for making an app this size, so I would focus on this first, getting the types working and typing everything. At least in the pure-JS side of things (see the Decoupling from Vue section). The longer you put this off, the harder it will be to add types later.

Avoid over-optimization

Some of the algorithms, which are already hard to follow, were made much harder by optimizing to avoid looping twice instead of once, etc. This is very detrimental to productivity. Algorithms are almost never the real bottleneck. Code at this level should be written as closely as possible to how we actually think of the problem and should follow our mental model closely.

Code reviews

I would honestly rethink the PR workflow here, it probably isn't necessary to have reviews at all. Let trusted core devs work on master together and communicate/coordinate to avoid complex conflicts, and just do PRs for third party submitters.

Don't avoid classes

Classes are not automatically bad. Inheritance is generally not the best solution, but often classes are a good and useful way to orgnanize the relationship between methods and the data they operate on. They also play well with browser tools and optimizations, making it much easier to inspect their data in e.g. a browser's devtools, than it is to inspect lexically captured values in a closure. FP and OOP are not better or worse, they'e complementary.

Use Prettier instead of StandardJS

I get the idea of wanting to avoid thinking about formatting and styles. And that's actually why I gave up on trying to format my own code and just let VS Code format it for me on save, which internally uses Prettier. This is subjective but StandardJS makes weird style choices and Prettier's defaults in VS Code seem much more reasonable and less strict.

Issue format

It doesn't always make sense to put things in problem/solution format. Maybe the person doesn't have a solution, or doesn't have time to think of a solution, or the solution is obvious and implied. Their feedback is still valuable and you discourage it by requiring this format when they can't adhere to it.

taoeffect commented 3 years ago

Thank you Steven for sharing your preferences on how this project should operate!

We use a problem/solution format because it allows us to discuss and focus on real problems, one at a time. It also lets us analyze whether something is in fact a problem (or not), and its potential solutions. If you don't have a solution, that isn't a reason to not open such an issue. Others might have a proposed solution that they can comment with, or it might turn out to be the case that the problem isn't a problem at all.

You've raised 14 different concerns here. If you feel strongly about any one of them, open an issue for it, and allow people who are interested to focus on it in that issue. At present, a discussion in this issue between all team members will be close to impossible, because there are so many totally different topics to discuss, and it would fill up email inboxes. For these reasons and more, we use the Problem/Solution format, and we ask that everyone adhere to it.

For example, if you believe that we shouldn't do code reviews, then open an issue for that, and tag it with Kind:Process. Then we can discuss what the problem is, and what the solution is.

Some of the suggestions you bring up very well may have merit to them. Perhaps Prettier is better than StandardJS. Who knows? This is something we as a team can discuss and collectively decide together. If you feel strongly about Prettier, open a Problem/Solution issue for it and tag it with Note:Tooling.

Some of these topics we've discussed before, for example, SBP, project configuration, project goals, the database, etc. We even came to conclusions on these topics, for example, with the database I thought it was clear that our solution is the simpler solution, and that we do not have any need or use for something like MySQL, which would not only make the project far more complicated (for no reason), but it would also make it much more difficult to protect user data. I will spare others a rehashing of the other subjects, but if anyone would like me to elaborate on them please feel free to post a comment below to the GH discussion or on Slack/Gitter.

Another reason we use Problem/Solution format is so that we can assign issues to people to work on. It's not really possible to assign this issue to anyone, because it amounts to a list of personal preferences that you'd like to see implemented. And a lot of your preferences I agree with! For example, I agree that business logic should be de-coupled from the UI framework. In fact we are doing that already in the gi.actions SBP namespace. As you mention, it will be usable from a CLI, and that is a rule-of-thumb we use. We even have an existing issue for it: #749

So, thank you for your thoughts and feedback! It is always appreciated because anytime anyone sees a way to improve the project, we're interested in hearing from them. However, we do ask that everyone respect our process, which isn't even ours per-se, because we adopted it from the C4.1 system invented by Pieter Hintjens from his ZeroMQ project. A lot of thought and reasoning has gone into every single decision we've made over the past 4 or 5 years by all of the team members, and newcomers to the project might not be familiar with that reasoning.

I'm going to close this issue because it isn't in the Problem/Solution format that we require, but that doesn't mean that discussion isn't welcomed. @sandrina-p recently pointed out that Github has a discussions feature that we can use to avoid filling up people's Inboxes with notifications. So maybe let's give that a try?

Feel free to continue the discussion there!

thesoftwarephilosopher commented 3 years ago

That discussion is not visible to non-team members.

taoeffect commented 3 years ago

Yeah that's an unfortunate side-effect of using GH discussions. At some point we'll have decent forum software for the project. If anyone who's not able to view that wants to join in, for now we can use Gitter/Slack.

thesoftwarephilosopher commented 3 years ago

Or we could just have this discussion here where everyone can comment. Maybe they could use the quote-reply feature to reply to just a subset of an idea, and if a bigger discussion starts to form, a new issue could be created for it. Just an idea.

thesoftwarephilosopher commented 3 years ago

However, we do ask that everyone respect our process, which isn't even ours per-se, because we adopted it from

That makes it your process and you have the freedom to adapt it to your project's unique needs and team members.

with the database I thought it was clear that our solution is the simpler solution, and that we do not have any need or use for something like MySQL, which would not only make the project far more complicated (for no reason), but it would also make it much more difficult to protect user data

I'm pretty sure the current database and SBP and GIMessage and Vuex situation is more complicated than a MongoDB + REST API solution for the equivalent data. And Mongo or any other NoSQL DB can be secured relatively easily. They have documentation on this, it's probably a full time week's worth of work to set one up securely with basic Node.js integration.

taoeffect commented 3 years ago

That makes it your process and you have the freedom to adapt it to your project's unique needs and team members.

True, it is our process now too.

I'm pretty sure the current database and SBP and GIMessage and Vuex situation is more complicated than a MongoDB + REST API solution for the equivalent data. And Mongo or any other NoSQL DB can be secured relatively easily. They have documentation on this, it's probably a full time week's worth of work to set one up securely with basic Node.js integration.

Well, it isn't more complicated, and there is absolutely no reason for us to be using MongoDB. It makes zero sense in our architecture. What we have now works great, it meets our project goals, it works better than MongoDB, and unlike MongoDB, it actually works for what we need it to do.

taoeffect commented 3 years ago

I can sense this issue is going to devolve into argumentation because of how it's setup to call everything in this project "wrong" without having an understanding of the choices we've made, and why, and without awareness of the existing issues that we have on many of these subjects (e.g. TypeScript vs Flow - an issue that can be picked up again if you want to take it on; CLI paradigm, etc.).

Let's not let that happen. I've unlocked the issue for now, but might re-lock it later if this devolves into arguments. In our Github workflow, people:

  1. Open individual issues for individual problems, and continue discussion there, or continue existing discussion in existing issues
  2. If they see something in this particular issue that they'd like to discuss, then in order to get greater clarity, they join us on Slack or via Gitter
thesoftwarephilosopher commented 3 years ago

Thanks for unlocking it @taoeffect . I think this issue is a good place to incubate discussions about some of the points, which can then be moved to new issues. I created it because it's all of the feedback that I have after over a month of working on the team, or at least all that I can remember. And it comes from many years of working as a successful software developer and working with very smart software developers and learning from them. So I wanted to put it here, but don't really have the time to work with a long and complicated process that doesn't really fit my feedback very well. Besides, I linked to this issue in the Slack, so it is already a good centralized place to start deeper discussions with the whole team.

taoeffect commented 3 years ago

Your feedback is very much appreciated, and my problem isn't as much an issue with the contents as it is with how we're using Github.

We are not using Github issues as a forum. Please respect the process we have in place for Github, which is for individual problems and their corresponding solutions. Until we get a forum, please use Slack, Gitter, or Github Discussions for forum-type discussions. Or, as mentioned above, open an issue, or comment on one of the existing ones.

thesoftwarephilosopher commented 3 years ago

@taoeffect If you want to continue our email conversation about the feedback I provided in this issue, please continue it here instead of in that email thread, and only if you agree that it should be added to another invoice. Although I am willing to continue to consult on this project in terms of advice through conversations, my limited time requires me to do it within time slots allotted for paid consulting work. We can use the rate agreed on in our last contract and I would add the past few hours of unpaid feedback including this issue and others in Slack. Thanks for your understanding.

taoeffect commented 3 years ago

@sdegutis We don't have a contract with you for consulting, so we would need to create one if we were to do that. However, it sounds like you're not particularly interested in the project, so let's just leave it at that. Regarding unpaid feedback, I thought you were volunteering feedback after you'd left the project. Apparently you were volunteering feedback and charging us for it. That's unusual, but nevertheless I've gone ahead and sent payment, along with a formal Stop Work order. Thank you for the contributions you sent in during your time with us.

sandrina-p commented 3 years ago

Just to let everyone know (for transparency), I commented on the Github discussions.

@taoeffect, I'd recommend updating the docs (or using Github Wiki) to explain the core choices made. That way everyone will have a clear understanding of the why behind those choices.

thesoftwarephilosopher commented 3 years ago

Our email conversations once again came back to this topic, so I'll answer you here:

Some of your goals need to change, to get the MVP out the door.

As those goals change, some other things should change too, like ripping out SBP and changing to MongoDB, to support the MVP goal, which should be the main one.

These and similar changes will take a little time in the short term, but significantly reduce your long-term goals of shipping the MVP.

Steve Jobs used to say β€œreal artists ship.” This is the point of this whole issue. Significantly reduce the MVP and adjust everything else accordingly.

Forgive my briefness, I'm offering this feedback freely outside of any allotted time, and have to get back to my many other duties.

taoeffect commented 3 years ago

Steve Jobs used to say β€œreal artists ship.”

Yes, shipping is good. πŸ‘

That is why we're wrapping up the prototype now instead of deleting everything we've done and starting over, which seems to be what you're suggesting, as that would take a lot longer than "a little time". There isn't that much left to do with where we are right now.

I think I've asked you multiple times now however to please not treat our Github issues as a forum. We can continue this conversation either over email, or on our Github discussions page, which is our forum for now (for developers only). Getting a larger forum for both development discussions and community discussions is something we will do soon too!

So, please refrain from re-commenting on this issue. If you can't, I'll be forced to lock it again. Thank you for your understanding.

thesoftwarephilosopher commented 3 years ago

I already told you I'm not going to discuss your project in private email threads anymore. I just don't have time for that. And like I said, your own team is full of brilliant and insightful developers, past and present contributors, who can offer better and more relevant feedback than I can. So feel free to discuss these things with them in your forums.

taoeffect commented 3 years ago

Apologies Steven, you sent me 9 emails today during our very long exchange on the subject, plus the above comment, so it seemed to me like you wanted to discuss this. Locking this issue now. If anyone wants to pick this up, please do so here, or on our Slack, or in our Gitter, or in one of the existing issues, or create a new issue if an issue doesn't exist.

We should have much better forums up and running sometime next year. This has been a challenging year for us, thank you for your understanding / patience, etc.

taoeffect commented 3 years ago

I have given this more thought, and have decided to unlock this issue because although this discussion does belong in a forum, the fact is, we do not have a good public forum at the moment. So while it remains the case that there is no better place to have these discussions (as @sdegutis points out above, Github discussions is unfortunately private), we can go ahead and use Github issues with the Note:Discussion label as our forum for now.

Thank you Steven for pointing this out! I do worry that by having forum-like discussions here, we might overwhelm people with emails, but while there are no explicit complaints about that, we can experiment with it, at least until we get a real forum up and running.

thesoftwarephilosopher commented 3 years ago

People can just click Unsubscribe on this issue if they don't want email updates from it.

Two things I had left to clarify:

  1. I'm not recommending rewriting your project. You have a lot of valuable assets. The Vue code is great, the design work and CSS is great. It's the piping in between these that's a bit rusty. It's like a completely messy and disorganized kitchen with no free counter space. It might just be quicker to clean it before finishing cooking the meal.

    So it might be quicker to take a little time to replace Flow with TypeScript, remove SBP, separate out business logic from view, make sure it's fully typed, replace the build system, etc. which might take a few weeks but will probably make your team much more productive.

  2. None of my advice goes against your bigger picture goals of decentralization, encryption, etc. For example, hosting a private NoSQL server next to the web server does not take away decentralization. You believe that it's less simple, but you previously agreed that it still meets the decentralization requirement. There are some implied goals you have, such as wanting to use Etherium concepts for the database, but those aren't needs.

    As I said early on, you have conflicting goals and if you prioritize them more explicitly, it will help you in many ways. My recommendation is that the MVP goal be highest, since I do not think anyone wants a protocol as much as an app, and that technological innovation is generally not compatible with writing an app except at a 1:10 ratio so that the app gets 90% of the focus and the innovation never takes away from the goal of shipping, especially shipping an MVP. I've ran into this problem so many times in the past 10 years and have had many failed personal projects because of it.

taoeffect commented 3 years ago

People can just click Unsubscribe on this issue if they don't want email updates from it.

Excellent point, I forgot that Github allows that granularity for notifications. πŸ‘

It's the piping in between these that's a bit rusty. It's like a completely messy and disorganized kitchen with no free counter space. It might just be quicker to clean it before finishing cooking the meal.

I think part of the reason it may appear that way is because we've developed a new framework for creating decentralized applications (#1002).

Normally frameworks exist outside of the app, but because we developed the app in tandem with the framework (to achieve our stated goals), it's as if you included the source to Vue.js / React / Ruby On Rails / MongoDB, or whathaveyou, inside of the app.

Now, our framework is considerably smaller than all of those other frameworks I listed, but it still adds substantial heft to the app, and since it's undocumented at the moment, it creates a bigger learning curve to onboard developers. These are issues we are very much aware of, and we have open issues for addressing this: #1002 #749 #430, and we may open up additional issues as well.

In spite of this, the app is still remarkably well organized. We did a fantastic job splitting up the app's components into their logical folder hierarchies, and once we split out the framework, the code will be smaller, and much easier to read and understand.

With respect to your concrete suggestions, I will address each individually here:

None of my advice goes against your bigger picture goals of decentralization, encryption, etc. For example, hosting a private NoSQL server next to the web server does not take away decentralization. You believe that it's less simple, but you previously agreed that it still meets the decentralization requirement. There are some implied goals you have, such as wanting to use Etherium concepts for the database, but those aren't needs.

A NoSQL database makes us fail to achieve our goal/requirement of having an app that offers real end-to-end encryption. We get nothing by switching to it, and we lose a lot in terms of both time spent and functionality lost.

thesoftwarephilosopher commented 3 years ago
  • remove SBP: SBP is a critical core part of the app and significantly improves our productivity and security, as well as our logging, and in the future, our documentation as well (issue #317). We will be only increasing our reliance on it, for example, possibly replacing Vuex with an SBP-based datastore (#588).

It doesn't improve productivity, I'm pretty sure it only slows it down. It isn't necessary to secure the app or add logging. These can all be done without introducing an implicit global module system.

A NoSQL database makes us fail to achieve our goal/requirement of having an app that offers real end-to-end encryption. We get nothing by switching to it, and we lose a lot in terms of both time spent and functionality lost.

MongoDB for example offers datafile-encryption and combined with secure-transport becomes full encryption in practice.

Sometimes time spent is not really lost but becomes a lesson learned, which is still very valuable. You also gain much more simplicity, easier ramp-up for new developers, and easier/safer maintenance for senior devs.

taoeffect commented 3 years ago

It doesn't improve productivity, I'm pretty sure it only slows it down.

SBP is a programming paradigm, so this statement is equivalent to, "I'm pretty sure [OOP/FP/etc] only slows down productivity." Perhaps you find it challenging, but I find it immensely valuable and rewarding, and so do others, as SBP-style paradigms have been around for decades.

It isn't necessary to secure the app or add logging. These can all be done without introducing an implicit global module system.

Anything can be done with anything. We could even write the app in assembly. The fact that something can be done differently isn't an argument against doing it a specific way.

MongoDB for example offers datafile-encryption and combined with secure-transport becomes full encryption in practice.

None of that is end-to-end encryption.

You also gain much more simplicity, easier ramp-up for new developers, and easier/safer maintenance for senior devs.

Yes, it's difficult to ramp-up new developers when you don't have documentation and when the framework is embedded into the app. I addressed this above.

thesoftwarephilosopher commented 3 years ago

SBP is a programming paradigm, so this statement is equivalent to, "I'm pretty sure [OOP/FP/etc] only slows down productivity." Perhaps you find it challenging, but I find it immensely valuable and rewarding, and so others, as SBP-style paradigms have been around for decades.

It's not really on the level of a paradigm. It's just an ad hoc module system, except that it encourages implicit dependencies, and is incompatible with type checkers that could catch typos in normally imported function calls.

MongoDB for example offers datafile-encryption and combined with secure-transport becomes full encryption in practice.

None of that is end-to-end encryption.

You're right that it's not exactly E2EE. So I guess I did advise against a firm goal.

So I guess I just disagree that E2EE is necessary. I guess it protects against the one situation where bad actors within a trusted organization are able to get access to data they shouldn't have. But it doesn't seem like a good goal to have all data completely opaque to all parties with legitimate authority to protect the common good. For example, criminals could end up using this software to store E2EE data that can't be inspected by the government for public safety. It needs to be inspectable by legitimate authorities.

You also gain much more simplicity, easier ramp-up for new developers, and easier/safer maintenance for senior devs.

Yes, it's difficult to ramp-up new developers when you don't have documentation and when the framework is embedded into the app. I already addressed this.

I suppose if you extract the database system and give it a nice wrapper API, this restores most simplicity at the app level. But it can't ever be as simple as a simple NoSQL database, because it has an inherently more complex design.

taoeffect commented 3 years ago

It's not really on the level of a paradigm. It's just an ad hoc module system, except that it encourages implicit dependencies, and is incompatible with type checkers that could catch typos in normally imported function calls.

It's on the level of a paradigm. It's not just a module system. It can be viewed as adding implicit dependencies, in the same sense that every time you add a key to an object the object gets a key and to access that key you don't need to import anything but you can simply access it. It's only partially incompatible with type checkers, and that can be fixed too.

So I guess I just disagree that E2EE is necessary. I guess it protects against the one situation where bad actors within a trusted organization are able to get access to data they shouldn't have. But it doesn't seem like a good goal to have all data completely opaque to all parties with legitimate authority to protect the common good. For example, criminals could end up using this software to store E2EE data that can't be inspected by the government for public safety. It needs to be inspectable by legitimate authorities.

You are welcome to have this argument with the makers of Signal, Matrix, GPG, PGP, Wickr, Bitcoin, WhatsApp, Telegram, Ethereum, Keybase, Bitcoin Cash, Namecoin, HighSide, ChatSecure, Status, Wire, FileVault, BitLocker, CryptPad, FlowCrypt, Ricochet, Secure Scuttlebutt, Zcash, Silence, Briar, Conversations, and probably dozens or hundreds or thousands of other apps and software. Indeed, it's a debate that people in a lot of industries have, even locksmiths.

From a technical standpoint, our goals eliminate MongoDB and similar technologies as viable.

But there is another reason that we've implemented our own protocol, and that is to be compatible with future tech like IPFS and Dat, if we ever decide to use them as a backbone, or if anyone else decides they'd like to run their own implementation on top of one of those protocols.

But it can't ever be as simple as a simple NoSQL database, because it has an inherently more complex design.

Sure it can, and I disagree that it's more complex. It's just different and you're not used to it.

thesoftwarephilosopher commented 3 years ago

We'll have to agree to disagree with all those points.

taoeffect commented 3 years ago

We'll have to agree to disagree with all those points.

πŸ‘