Technical Requirements

[grantcodes]

Hey all,

I figured it would be good to start a community project by defining some goals, which can then be spit out into separate issues at a later date. I will keep this list at the top updated based on feedback.

This thread is for code and implementation related thoughts. Meta requirements are in #1

• [ ] Accept all types of micropub requests (micropub.rocks as a base)
• [ ] Media file handling
• [ ] Normalize & create valid mf2 json for any create requests
• [ ] Full test coverage
• [ ] Express compatible (maybe others koa etc)
• [ ] Fully extendable to work with any existing or future project
• [ ] Media endpoint (optional)
• [ ] Some form of logging
• [ ] Alt text support (+normalisation)
• [ ] Handle scopes

[miklb]

I'm not sure of it would fall under technical requirement but support for alt text on images is a high priority for me.

[grantcodes]

So I did a bit of digging into various modules used by the majority of micropub endpoints:

Core Dependencies

Module	Use	postr	micropub-express	indiekit	mastr-cntrl
body parser	Parse requests	☑	☑	☑	☑
express	http server	☑	☑	☑	☑
multer	Handle files	☑	☑	☑	☑
node-fetch	http requests	☑	☑	☑	☑
cors	accept cors requests	☑	❌	❌	❌
ajv	json schema validation	☑	❌	❌	❌
typescript	type checking	❌	☑	❌	❌

Dev Dependencies

Module	Use	postr	micropub-express	indiekit	mastr-cntrl
chai	testing	❌	☑	❌	❌
coveralls	test coverage	❌	☑	❌	❌
eslint	linting	❌	☑	❌	☑
husky	validation on git hooks	❌	☑	☑	❌
mocha	testing	❌	☑	❌	❌
nock	mock server	❌	☑	☑	❌
sinon	mock data	❌	☑	❌	❌
supertest	http testing	❌	☑	☑	❌
ava	testing	❌	❌	☑	❌
xo	linting	❌	❌	☑	❌
verror	fancy errors	❌	☑	❌	❌
bunyan	logging	❌	☑	❌	☑
@timberio	logging	❌	❌	☑	❌
@timberio/winston	logging	❌	❌	☑	❌

So an obvious conclusion in that the top 4 modules are a definite best idea for core functionality.

Outside of that they quite varying. But clearly @voxpelli and @paulrobertlloyd have put care into their testing

[grantcodes]

So based on #5, I took a quick first effort of splitting out my post type discovery code into a separate module. But I was more focused on testing out what sort of modules we can use between shared projects (the tech stack more or less)

I based decisions on the above tables and what is most popular & up to date on GitHub, so here is what I came up with:

Node version

I decided to use the LTS node version (currently 10.16.0). Anything above that should run fine too, but tests and whatnot should be run in the LTS version.

I will also note here for frontend packages I added a browserlist to the package.json to support the last 2 versions of current browsers by default.

Code format & linting

For this I went with what seems to be popular practice right now. So linting is set up with eslint following the airbnb config which is very reasonable imo. I also included a few other eslint plugins to watch out for some other best practices.

As well as that I also have a prettier config set up to match those eslint rules. So if a developer has prettier set up on their machine it will help auto format code to match the set standards.

Testing

There was no clear favourite testing framework from the existing modules so I went with jest as it seems to be the most popular, it is unlikely to be abandoned anytime soon, and it also basically includes all other pieces needed (assertions, code coverage, etc.)

I think for some extra niceties I will also set up coveralls to generate the code coverage badges and husky to run tests before pushing code.

Outside of that I think each project would need other helpers for testing and those can be chosen on a per project basis.

Logging

I'm no expert in this, and it basically came down to a tossup between winston and bunyan, and I chose winston purely because it is more popular. But I think there should be a rule that any project that uses a logger should allow it to get replaced with the users own logger (as long as it is compatible).

Typescript & other transpiling

I decided to stay away from any typescript or other transpiling type stuff for now. As I think the accessibility and ease of understanding in community projects and outweighs the benefits of transpiling.

---

That's about as far as I have got so far, but I think that would be a good base set of tools to build off of.

I'd love to hear if anyone has any opinions or preferences regarding anything I've mentioned.

[voxpelli]

@grantcodes typescript is not a core dependency of micropub-express, it's only used to do "type linting" through the use of JSDoc comments etc, so it should be treated as a dev dependency. I'm not transpiling anything.

Node version

+1 on targeting Node LTS, usually all of them, but since Node 8 is set to end its LTS within 6 months it makes little sense to target new modules against it

Linting

I would go with StandardJS as the coding style, in the flavor that uses semicolons: https://github.com/standard/semistandard One could go with Prettier as well if one really just want consistency and don't have to care about coding standards.

On top of that I would add other checks: https://voxpelli.com/2016/07/better-handle-npm-modules/

Though: I wouldn't want to push this as an important aspect. Important is the interoperability of things and coordinating efforts, not ensure that we all use the same coding standard and linting. Plurality is good not just in the IndieWeb as a whole but also in subcommunities within it 🙂

Testing

Same as previous one, I don't think this is an important thing to sync, whatever works will work and whatever a main contributor prefers they should use.

Logging

I don't think one should enforce any logging library, hence why my code uses my bunyan-adaptor as it exposes a minimal subset of bunyan and links that to either console.log or custom provided methods and allows the insertion of any other library that shares that same subset (or one can oneself use bunyan-adaptor to create such a subset of course)

That way one doesn't impose ones choice on anyone else but rather allows them to chose whatever they want.

Transpiling

I'm no fan of transpiling either, I think that just complicates things. I do like exposing type data, but that's possible without doing any transpiling.

[paulrobertlloyd]

My 2¢…

• Node: +1 on supporting LTS
• Linting: Pretty much what @voxpelli said. As long as there is linting, and the necessary files are in place to ensure incorrect syntax is highlighted in IDEs, etc. then I’m happy with whatever.
• Testing: I’m still wrapping my head around testing, and the various tools and techniques to help with unit testing code, so no strong opinions here. AVA seems nice, though.
• Logging: Also something I’m still trying to fully understand. I went with Winston only because the project appeared to be more active than Bunyan.
• Transpiling: As someone still fairly new to JavaScript, the less stuff like TypeScript the better, as it only serves to confuse. ES6 syntax (fat arrows,const/let`, etc.) is fine – it’s very much part of the language now and supported in NodeJS.

[grantcodes]

I'm fine with using the linting standard you suggested @voxpelli, I think it's pretty similar anyway.

But I don't think npm module checking stuff you suggested is really necessary, as long as a .package-lock.json is included I think that should be sufficient.

I also don't really understand the advantages of the bunyan-adaptor - If someone wants to use their own logging they still have to use something compatible with the bunyan methods and if they don't include one they don't get the advantages of nicer logs. So why not just use the nicer logs by default but still allow people to use their own compatible replacement?

But I do agree that not everyone needs to use the same testing / logging. But it would be nice if any module that ends up on the indieweb github org or @indieweb npm org would be standardized.

[voxpelli]

• package-lock.json isn't really useful in libraries + it doesn't do all the checks that dependency-check and installed-check does.

  • dependency-check checks whether one is using a module in code without having added it to ones package.json (flattening can have made it available for you anyhow) and whether one claims to have dependencies that one no longer has
  • installed-check checks whether ones supported node version ranges is the same or narrower than that of ones dependencies. Useful as it would be hard to claim to support a wider range of node versions than what ones dependencies does.

• The size of bunyan-adaptor is minimal and it outputs human readable logs.

  • The size of eg. bunyan is not that minimal (it even on some platforms installs native dtrace extensions which makes it quite a slow one to install) + the output of bunyan is JSON which isn't that user friendly unless passed through the bunyan cli tool.

  • To me "nicer logs" is a decision far out of scope of any individual module. If they enable one to add it when one wants, then great, and that's what bunyan-adaptor does, but if they starts to have opinions in how stuff should be logged and starts to do fancy logging – then that I don't like.

  • With something like bunyan-adaptor one can standardize on an API without having to take a stand on whether Bunyan, Winston or plain old console.log() is to be prefered.

[grantcodes]

Cool thanks for explaining that @voxpelli

If the checks are just dev dependencies then no harm in including them.

R.E. logs - How about we say something along the lines of if community run modules include logging, they should be extendable and use the bunyan style (preferably by bunyan-adaptor or something lightweight, then I think people can use whatever they want as I think everything mentioned is compatible with the bunyan methods.

[vipickering]

Sorry I've had a lot going on in my personal life the last few months and not had chance to catch up on this.

I wouldn't use Mastr Cntrl as a model for anything, it's pretty poorly thrown together so I could learn more JavaScript/Node and IndieWeb stuff (hence the lack of testing etc). I always intended to rewrite it once I had made all my mistakes, its basically a working prototype held together with a lot of sticky tape.

I am not really fussy over tools to a point. My view is always that if it does the job, doesnt get in my way and requires less maintenence than the output it provides it is probably good enough.

For logging something bunyan style sounds good to me and widely understood. For testing, in the past I've used istanbul for unit tests usually with something like Mocha or Chai. For linting I prefer to stick to raw JS nothing fancy like Typescript and things which are all the rage, especially since you get most of the latest and greatest JavaScript on Node. So I just stick with eslint.

The only big gotcha I have found so far (and something you may want to think about in the future) is when I moved to New Zealand I switched timezones. This meant I had to tweak MC to be in NZ time and my Jekyll website. I had a fun scenario where I would post to MC (or get sent a webmention) then it would sit and wait for 13 hours before it hit the UK time and the post happened, or post immediately from MC then output everything in UK time on the website. I was storing timestamps in a few places and it took a while to track them all down. It gets made worse because you are dealing with a minimum of two sites talking to each other, each potentially in different parts of the world or with different TZ settings.

My personal time is very limited in the immediate future but I'll do what I can to chime in or contribute when I can :-)