Using a DB (mongo?) to store, access and update messages, resources, users, etc.

[robinfla]

Right now, node-json-db module is used as a simple database and it works for the simple purpose of storing team/user IDs and checking if said user has accepted the ToS.

However, the module is fairly limited and the JSON file could be a mess if we add the messages and resources. If not, those files will have to be scattered and divided into subfolders of subfolders, which is not ideal either. There's probably something better to be done here.

I've only used mongoDB for now so that's what I'm thinking about.

What's your opinion on this matter?

[railsstudent]

Eventually we have to move the messages and data to mongodb but we are in early stage of the project and may want to encourage as many people to help out, even beginners. Having low learning curve seems more important than laying down all the infrastructures. From my experience, this project needs fair amount of understanding of JavaScript, ES6, Node and Slack Web API. Adding MongoDB and NoSQL to the pile may overwhelm newbie developers and open source. Heck, I am slightly overwhelmed when setting up slack workspace, bot and ngrok in local machine.

[robinfla]

I totally agree with you. Same thing happened (is happening?) to me while trying to run my local environment. However there may be opportunities to differentiate projects/issues that don't need database skills from projects/issues that do. We were actually talking about that with @BethanyG on a broader subject: onboarding people who want to collaborate by teaming them up on small projects or pointing out "easy" issues

[angelocordon]

Great discussions @railsstudent @robinfla - while I see we'll eventually need Mongo, I'm going to tag this as backlog for now so we can revisit it at the proper time.

[ajparise]

Just a random thought, what if the DB framework dependency was pushed to the outside instead of being tangled inside the rest of the code?

That way the framework could be swapped out without changing dependent code.

Refactoring this:

const db = new JsonDB('./app/src/app/users', true, false);
const initialMessage = (teamId, userId) => {
  let data = false;
  // try fetch team/user pair. This will throw an error if nothing exists in the db
  try { data = db.getData(`/${teamId}/${userId}`); } catch (error) {
    console.error(error);
  }

  // `data` will be false if nothing is found or the user hasn't accepted the ToS
  if (!data) {
    // add or update the team/user record
    db.push(`/${teamId}/${userId}`, false);

    // send the default message as a DM to the user
    message.channel = userId;
    const params = qs.stringify(message);
    const sendMessage = axios.post('https://slack.com/api/chat.postMessage', params);
    sendMessage.then(postResult);
  } else {
    console.log('Already onboarded');
  }
};

Into something like this

const db = new UserDB();

// Set logging callback
db.log = console.log

const initialMessage = (teamId, userId) => {
  if (!db.hasUserAcceptedTos(teamId, userId)) {
    // send the default message as a DM to the user
    message.channel = userId;
    const params = qs.stringify(message);
    const sendMessage = axios.post('https://slack.com/api/chat.postMessage', params);
    sendMessage.then(postResult);
  } else {
    console.log('Already onboarded');
  }
};

[d3vild06]

@ajparise what you're referring to here is the facade pattern which is a great approach overall. It's a little more of an advanced topic for most but I don't think we should shy away from implementing it.

I would, however incrementally introduce this pattern once we actually implement MongoDB (or another DB) first. If we go with an ODM like Mongoose, it already provides a decent API that can simplify the DB calls and have the code be more readable.

@railsstudent @angelocordon @robinfla I'd be more than happy to collaborate on implementing MongoDB into the project.

[BethanyG]

@stain88 and I have been kicking this around as we get the /resources architected. We really are on the verge (just look at that development branch file tree and do the math!!). I have been trying to mostly stay out of the discussion because I have a strong preference for a DB - but understand that introducing one adds complexity and intimidation (both in setup and in coding) to those not familiar. I'm also far more familiar with relational DBs (Postgres, MySQL, SQLite, and the like), so that biases me.

@ajparise @sergeant-q @railsstudent @robinfla @stain88 - given our current situation & file tree with /resources (and the need to quickly add more resources, along with projects and other things we're thinking of) - what does adding a DB look like?

Is there a way to do this work in parallel, or does it require ripping things apart again?

Is there a way to modularize the DB-dependent code so that there's meaningful areas where we could still make newcomers both productive and welcome? While our file tree is frightening, it's a whole lot less frightening than query-writing is to many people. In addition, making yml files for resources and playing with templates is a pretty beginner-friendly task.

There is also the question of how we get the sheer amount of content into the system efficiently (see #9) in the first place. Because (in the case of individual resources) they still need to be written out in Slack markdown in some shape or form and then inserted into a message template.....

And resources are of course an entirely different thing than the storing of users who've acknowledged the COC, and other information we'd be storing in the DB.

I'd personally would love to see it happen, and I'd love to have more people working on this project. So - thoughts? Feelings? Volunteers?

ok. Stepping down off the soapbox.....:-)

[stain88]

Fairly confident a DB can be implemented in parallel - once it's set up, we can probably use a script to drop all the yml's into it.

Regarding #9: If we have a DB, it would need to be hosted somewhere, right? If it's hosted, then there will be an API to connect with it, and if there's an API, there can be a site associated with it. I know RoR has an easy gem that adds an /admin url to a site, but maybe something similar can be done in Meteor on codebuddies.org? Like, codebuddies.org/resources checks status, and then that can be used to make POSTs/PUTs requests against the API.

[ajparise]

@sergeant-q Yep, a facade-adapter combo, leveraging a ports-adapters architecture. We'd have one port (facade) and multiple adapters for each DB we need to support.

I would, however incrementally introduce this pattern once we actually implement MongoDB (or another DB) first.

Alternatively, we can create an adapter for the existing database and incrementally introduce the facade code. Then we'd just need to create an adapter for whatever DB we choose in the future. The main purpose of using this pattern upfront is to simplify and separate the process of swapping to a different database by de-coupling the code from the database layer.

@BethanyG

Is there a way to do this work in parallel, or does it require ripping things apart again?

We could prepare for the change before we even decide which framework to use. That's the advantage of ports/adapters, it mitigates the vendor-lock in on frameworks. After the pattern is in place, moving to a new DB would be completely isolated from core business logic.