search

jfhbrook / node-ecstatic

A static file server middleware that works with core http, express or on the CLI!
https://github.com/jfhbrook/node-ecstatic
MIT License
975 stars 194 forks source link

ecstatic's Past, Present and Future #259

Open jfhbrook opened 5 years ago

jfhbrook commented 5 years ago

TL;DR - The most valuable thing ecstatic can do for the JavaScript ecosystem is to self-destruct, never to cut a new release again.

2009

I once had a friend tell me that for many people, the second or maybe first thing that someone gets really into, usually in their early 20s, becomes a memory like what people have about punk rock in New York in the '80s. "We were There and we were Doing It," these people would say.

We were of course talking about Node.js. In the neighborhood of 2009 I was in my early 20s, and in fact Node.js was about the second thing I really got into (after webcomics in the mid-aughts as a teen). We were certainly There - with perhaps 40 people in the #node.js irc room around then - and we were definitely Doing It - nobody had written any of the node libraries yet and the possibilities seemed infinite. Someone ported Sinatra to Node.js, someone hacked up a baby package manager on top of CouchDB, a friend of mine wrote a huge mess of utility modules and me? I learned Node.js as a hobby while working on my engineering degree.

It was just a few years later that I got my first job out of college with Nodejitsu. Nodejitsu is probably most famous for being an enterprise npm host before npm Inc was founded in 2014, having incredible journeyed shortly after, but when I worked for them in 2011/2012 they were actually supposed to be the next Heroku. I was an early fan of their platform and to this day I will maintain that the ux of their cli tool jitsu was solid. Mostly due to my association with substack I'm sure, they agreed to hire me as a support engineer.

It was at Nodejitsu that I wrote ecstatic. Over the summer of 2011 the team decided that the way to reach market dominance was going to be building and open sourcing a web framework, called Flatiron, and by the time fall was in full swing they cut an initial release.

Being a support engineer meant I was largely responsible for fielding questions and issues that people had with Flatiron. There were many. A lot of this was understandable given the relative age of the framework: A module had an edge case nobody thought of, or someone had fat fingered a poorly tested procedure and shipped a bug. Often, I would be the one to issue fixes.

One of the things that Flatiron was missing that a lot of people complained about was a static fileserving component. As far as I know Flatiron never shipped one! I think they expected people to use cloudhead's module but, while apparently rfc 2616 compliant, it wasn't written with middlewares in mind. So, I decided to try to solve the problem by doing it myself.

My initial version was really bad - a 30 line call to readFile - and I needed a lot of help to get it working. Commits through 2012 were mostly me and substack. I found an ally in him on this project: I needed something to hand to people trying to use Flatiron, and substack wanted to drop express. We worked together and shipped something.

Nodejitsu never adopted ecstatic as their own. After my time at Nodejitsu, for better or worse, ecstatic remained my responsibility alone. In the end, Flatiron didn't gain an appreciable amount of market share and in fact ecstatic eventually dropped Flation support.

For the past 7 years or so, ecstatic has largely been a solo operation. That's not to say that I haven't received any patches in this time, and in fact you can see a whole list of contributors due to someone at Google telling me that I needed to give them partial copyright. However, I've been the primary reviewer and gatekeeper on all external contributions and otherwise the primary developer of new features.

npm

On April 23rd, I received a stern letter from npm support. As far as I can gather, this is the series of events that transpired:

  1. Somebody found a security vulnerability in ecstatic.
  2. This person saw the "Report a vulnerability" button on npm's website that actively encourages people to report me to npm's security team instead of directly.
  3. npm support downloaded my module and spent a bunch of time to verify the bug report but stopped just short of giving me useful information about fixing it.
  4. npm sent me a letter threatening to issue a "security advisory" in 40 days.

You may recall the case of left-pad. In case you missed the story in 2016, it went like this: A developer registered a project on npm named "kik", a labor of love open source framework. One day, a corporation came along and demanded he change his project name. When he objected, npm gave him no choice in the matter. Presented with this decision, he ragequit and had npm delete every single one of his modules from npm. This broke a whole bunch of code because other projects were depending on his work. npm, in a controversial decision, restored the specific package@version being used by babel and other breaking tools.

As a developer, I at the time sided with npm. After all, this guy threw a temper tantrum that broke a bunch of stuff for everyone! His packages were open source, so npm was legally free to host them. He didn't have a leg to stand on!

But it also shows how little control you have when you use the npm platform. They're free to modify your packages at any time without your consent, while refusing to remove data when asked. When this company came down on this guy, npm left him out to dry and there was nothing he could do about it.

I think my recent problems have further highlighted some of these issues with npm. It's not that I don't take security seriously - I think it's important that people can't get hacked - but on a certain level I definitely felt like npm was shaking me down for free work, and it didn't feel good.

This is what happened next:

  1. I verified their report and wrote a patch for the v4 branch, and published it.
  2. In response, npm issued this security advisory.
  3. Partially due to legitimate concerns about people installing dangerous code and partially due to frustration with npm, I asked them to unpublish all of the affected versions of ecstatic. I was fairly certain that they would tell me no.
  4. By the time I realized that npm was already creating very ugly severity warnings on install, it was too late and npm ran the unpublish. I was surprised they did it, but wasn't exactly mad either.

This is where http-server and mime come in.

http-server to this day remains the primary connection to Nodejitsu. This package was and is primarily intended to be used as a development server, and these days is ultimately a thin wrapper around ecstatic. Funny enough, ecstatic ships its own cli component, but inertia and historically better docs win here.

mime is a simple package that has somehow caused me, and now the http-server maintainers, a whole mess of grief. ecstatic uses mime to intelligently set the content-type header in responses. The mime module went through a major rewrite a few years ago, and dropped support for .types files, which I believe are an Apache thing. Either way, the upgrade path was fairly painful but eventually happened about a month ago.

Unfortunately, http-server is a little behind on the upgrade and it turns out it's a bit of a beast - an honest surprise. The ramifications of this are that the http-server team's package broke when npm unpublished the ecstatic versions in a way that couldn't easily be fixed by an upgrade on http-server's end. Unsurprisingly, people were pretty angry.

Open Source

Over the years, I've become more and more disillusioned with open source software, and these recent events with npm have been a pretty consistent demonstration of a number of the issues I've had.

When I was a teen and first learning about Linux, one of the major things I read about was the difference between Free Software and Open Source Software. Free Software - free as in Freedom, not beer - was software that was supposed to be protected from corporate interests by ensuring that modifications to source code would by license need to be shared with the world, and came from many of the frustrations of working with proprietary, properly closed source software over the years. Open Source, on the other hand, is concerned with a different flavor of freedom. It includes the freedom to not think very hard when hacking on cool side projects sure - and I think this is why I liked using MIT licenses in 2011 - but crucially, it also includes the freedom to use and modify someone's code for internal use without contributing those changes upstream. This is by design.

This manifests as a freedom to exploit. Choosing a license like the MIT license becomes sort of a Devil's bargain: Your project is significantly more likely to get used by other people and become low key famous, but because nobody has an obligation to help you, they won't. From a business perspective this makes total sense. From my perspective: nobody ever paid me to work on ecstatic outside my time at Nodejitsu, and in fact nobody was ever paid to work on ecstatic long term.

In general, open source is pretty thankless. People complain a lot and nobody helps you. Maintaining a non-famous non-sponsored module is lonely and exhausting. Often, somebody will come by with not a suggestion but a complaint, and then get huffy at me when I don't have time to work on it. People don't seem to understand that not only do I typically work a full time job but I'm also extremely non-neurotypical in a way that significantly decreases my spoons when things are bad. This doesn't even mention life events that get in the way: just this week as I've been dealing with these issues, I lost my job and had to attend a funeral.

Meanwhile, I hear about people making bank off of my free labor:

image

and see them get angry when more free labor is denied [1]:

image

It's especially funny to me when people think I'll be offended by them using someone else's module. Oh, not my problem anymore? Sick!

But this is what I mean. When working on open source, people expect things from you for little reward, and eventually the open source maintainers that weren't able to monetize their work burn out and become proper leftists. It's often not a matter of if, but when.

Modern JavaScript

Punk rockers get old and find real jobs just like everybody else - "I didn't sell out, I bought in!". Node.js has evolved from a cool hack project in 2009 to a standard part of JavaScript development both on the backend and the frontend. Companies like NodeSource and NearForm have grown into well-respected and lucrative consultancies. Node.js itself became managed by The Linux Foundation, with sponsored seats at the table. npm became a real business. Many a thought leader from the old days has made a career out of getting paid to sound smart in meetings, bouncing between gigs as their employers start to realize they're being had.

I've also observed the Node.js community get more and more toxic in that time. I've seen the TSC get filled with white men who unironically support the meritocracy and I've seen them use shitty tactics to silence women. I watched the IRC room become less and less moderated to the point where I was legitimately worried about right-wing trolls, and when I tried to cause movement on the issue I got accused of having a mental break. More than once, I've watched corporate owners of open source projects fail to hire active unpaid contributors to those projects. I've seen multiple women abused by famous node people. All these issues make "JavaScript Is Fun And So Node Is Fun" less and less true by the day.

Mikeal observed that what counts as good code changes as well, and this is further complicated by the fact that I was an abysmally bad junior developer in 2011. Not only are there strange issues with the API - multiple ways of spelling each option and half-baked error handling being some of my favorites - but the code quality is also quite simply bad. Structurally I find the code pretty confusing and I blame it for issues such as this edge case around 404.html files and directory listings. In many ways, ecstatic isn't a very good base on which to build a successful static fileserving component. In another universe, I would have probably rewritten it from scratch.

As it stands, I see myself drifting away from Node.js and the code I've written in it. I started writing Python as my daily driver in late 2016, something I've been very happy about. I stopped working on wzrd.in years ago. I left the IRC room in a huff after my concerns weren't taken seriously. I've fallen behind on what's now considered idiomatic ecmascript. I'm beginning to feel a little like an aged rock musician with a small and aging fanbase and a boring, regular life removed from the heyday - a little like J Mascis buying a VW Golf.

But I kept maintaining ecstatic, the last project connecting me to who I was in 2011. Despite all of the issues I've had with the Node.js community, with open source, with angry people demanding things from me, with all the open issues I couldn't close because of missing spoons, I kept shipping small fixes and merging people's pull requests. Now, with these recent developments with npm and the security advisory, I've finally decided I've had enough.

Goodbye

Here's the plan.

Here are some suggested migration paths to mitigate this:

Finally: A thank you to everyone who has used ecstatic in kindness, to all that have sent me encouraging words, and to everyone who has sent me patches over the years. Up to this point, I would have said that you've made all this work worthwhile.

So long, and thanks for all the fish.

[1] To be fair to this poster, he eventually thought better of things, edited his comment to be less sassy and left some nicer ones. However, screenshots are forever, and even nice people can say mean things sometimes.

jhpratt commented 5 years ago

Are you willing to turn control of the library over to someone else?

adierkens commented 5 years ago

@jhpratt

I'll be available to answer questions about ecstatic as long as they aren't about re-publishing or patching older versions of ecstatic. I'm also not interested in a project transfer.

Doesn't sound like it

zunsthy commented 5 years ago

sigh :(

When working on open source, people expect things from you for little reward, and eventually the open source maintainers that weren't able to monetize their work burn out and become proper leftists.

ryangrahamnc commented 5 years ago

Should've contracted with the multimillion company, doings fixes for a decent six figgies, and close sourcing what you made. Then contract with every other company that aggressively wants a fix. Could have made a pretty penny.

strugee commented 5 years ago

Just want to thank you for being open and honest and working hard on ecstatic. I think it's really valuable to hear this perspective, especially from someone who does maintain a widely-used module. I'm sorry things turned out this way and you deserve better. For what it's worth, I've been a happy user of ecstatic for my personal site's local development server for years and I really appreciate being able to use it for so long.

I'm also curious if you think you'll be using the GPL more in the future? Or just releasing less (Node) software? Either way best of luck in whatever your future holds, and I hope the life issues clear up soon. Take care of yourself <3

MicahZoltu commented 5 years ago

Patches for the security vulnerability have been applied to versions v4.1.2, v3.3.2 and v2.2.2. Older versions will remain unpatched. I apologize for the inconvenience.

Are these going to be pushed to NPM before you check out? Looking at the package on NPM, only 4.1.2 of the ones you listed is published. https://www.npmjs.com/package/ecstatic

jfhbrook commented 5 years ago

I definitely published them to npm, I don't know why they claim otherwise. You'll have to take it up with them.

dotnetCarpenter commented 5 years ago

Finally: A thank you to everyone who has used ecstatic in kindness, to all that have sent me encouraging words, and to everyone who has sent me patches over the years. Up to this point, I would have said that you've made all this work worthwhile.

@jfhbrook Thank you too! You have always made it worthwhile to hack on ecstatic.

So long, and thanks for all the fish.

Well said 😄

I'm surprised but fully understand your sentiment. I'm gonna miss having a web server to hack on (as opposed to configure e.i. nginx and apache) to validate front end ideas.

Sorry about not finding time to fix the mime type stuff. It has been in the back of my mind months (or years perhaps).

Cypher1 commented 5 years ago

@jfhbrook Thank you for your years of service. Your work has helped me multiple times, both in my personal and professional life.

iamvijaydev commented 5 years ago

Thank you so much for all the hours you spent happily or otherwise on this project. I wish you all the best wishes to you and your close ones.

Arlen22 commented 5 years ago

Just thought I'd mention that it's really simple to serve a static directory with the send module, which is part of the express.js ecosystem, I believe.

const serveStaticFolder = (req: IncomingMessage, res: ServerResponse, pathname: string) => {
  send(req, pathname, {
    root: pathResolve(serverConfig.staticPath),
    index: ['index.html']
  }).pipe(res);
};
//this line is untested.
new require('http').Server((req, res) => { serveStaticFolder(req, res, urlParse(req.url).pathname); });

Just thought I'd mention it. I don't know why so many people are so enraptured with clogging up their file-system with wrapper modules and the like. Sure you have to write your own directory index generator, but I already did that quite easily, and it felt like writing PHP. So if someone gives you grief just send them that. That's pretty much my philosophy.

Thanks for your thoughts, I have some MIT-licensed projects myself, and this gives me some good things to think about. I also work on private software for pay, but that's unrelated to my free projects. It's a really crazy world. Sometimes I don't understand this whole thing about white men vs women, because I am a white man (although still young) and I value women equal to men. But then again, I have not even set foot on the forums your mentioned, or pretty much any forums at all (since I pretty much rely on API reference docs to get my work done), and so I don't have any experience in that arena.

mk-pmb commented 5 years ago

Wow, very unfortunate news. I can totally understand your frustration. Already in the leftpad case, my stance was that people are responsible themselves for setting up caches and mirrors of their dependencies, especially if they are a big corp. But yeah it is effort and yeah in a business that means it costs money upfront, before disaster strikes.

Thanks for having made and maintained ecstatic all those years. It had helped me a lot when I tried my first Node.js projects.

mwufi commented 4 years ago

wow thanks, i never realised :)

fabiosantoscode commented 4 years ago

Thank you very much for your work!

:heart:

jorenbroekema commented 4 years ago

I very much appreciate the story you wrote up helping us understand why these things happened, and shining a light on some issues of open source.

Thanks for your work, hope you can rest a bit now that you've made the decision to pull the maintenance plug, seems to me like you deserve it man!

tchakabam commented 4 years ago

Puh. What a blast. I only realized this had happened just a few days ago.

What I think: There is probably still a big problem with many of those "infrastructure" components! It reminds me that SSL lib issue years ago, where suddenly people realized half the internet is encrypted with a lib maintained by two people :D

For anyone looking for a drop-in replacement (rather coming from http-server), we are going with ws now, and I am rather satisfied (as part of the broad local-web-server package): https://github.com/lwsjs/local-web-server

And also, node-static (i.e the "cloudhead package" as mentioned here above of which I happen to also be a decent minor contributor) now also supports middlewares as shown in the current readme (by hooking into the native http package, so from that layer on you can support any middleware that builds upon that): https://github.com/cloudhead/node-static ... and it relies on the great mime package as well. Hope this is valuable info for anyone needing to move away from this package here.

@jfhbrook I hope you will be able to get back into open source community at some point, since you have been such a great developer and contributor so far. Hope you will be more successful in the future in that case to also see the products of your work as a business (of which you eventually will then also prosper and profit) and not just as punk rock 🎸 🤘 ;) Muchos kudos to u and cheers to NYC.

tchakabam commented 3 years ago

If anyone still looks for a replacement, node-static has a new maintainership and many issues resolved and PRs being merged: https://github.com/cloudhead/node-static/issues/224