Is this project dead now?

[bryantbiggs]

☝🏼 - honest question. trying to decide if I should fork or whats going on

[AndrewFarley]

@bryantbiggs Sorry, I think @bsamuel-ui is afk at the moment, and I'm quite swamped as well. But I'm around and just saw this via Slack, and will turn back on my notifications for this repo. Can I ask for assistance perhaps in deciding which pending MR's need approving so I can cut a release? There seems to be a lot of great ones in there. Anyone offer to help sift through them with me?

[AndrewFarley]

Oh, man it looks like I lost or no longer have write-access to accept merges. :( So... we're at the whim of others. @dschep / @bsamuel-ui around to help grant me or someone else maintainer privileges?

[bryantbiggs]

to start, getting some basic dependencies updated to patch vulnerabilities would be good (shameless: https://github.com/UnitedIncome/serverless-python-requirements/pull/586). outside of that, how can I help?

[dschep]

Fair question @bryantbiggs! I initially wrote this when at UnitedIncome, and continued to do some maintenance/mgmt because I was Serverless Inc's Python SME, but I haven't done anything with this in quite a while now (I do still use it on a side project here & there, more on that in a bit).

So.. that is to say, I'm not sure. I dunno if UnitedIncome is using it internally still(cc @dwolfand), but if they are, it's probably like how it was used when I was there & wrote this plugin(and as I alluded to, is how I still occasionally use this plugin on side-projects). That is to say: we don't use most of the features.

I initially wrote this because I wanted a solution simpler(and not tied to wsgi) than serverless-wsgi. We made it open source, then we accepted PRs from the community to try and keep the sls python plugin ecosystem from being fragmented. But now, that means there's now plenty of code in this plugin we're not super familiar with, and don't use.

@AndrewFarley, you & @miketheman have Triage permissions now it seems. Not sure when that got downgraded. I seem to still be an admin so I could change that, but I don't want to step on @dwolfand's toes since this repo is in the UnitedIncome org.

[bsamuel-ui]

UI is still using it internally, and I think I've got similar problems as @dschep, there's stuff we just don't use, especially the Windows support. Let's chat about making this be not dead, especially getting a release out the door sounds promising. @bryantbiggs I've sent you an invite to the serverless-contributors stack, can you check the #python-requirements channel>

[bryantbiggs]

thanks all! I use this in nearly every project (unless a client makes me use node), so I am very appreciative of what you all have provided here. And yes, I can understand that as the maintainers there are some hard decisions to make. If I had to fork the project, I'll be honest - Windows stuff would go right out the window immediately. No offense to Windows users, and I wish NPM was able to show stats on breakdown of which OS users are using, but I feel like that packages like this where its probably 5-10% of the user base is using something that is disrupting the flow/progress for +80%, that maybe its a hard decision that needs to be made where Windows support is dropped and its up to that userbase to take a fork and run with it if they so choose. Anywho, just a thought - I'll checkout the Slack invite and thanks for all the efforts!

[AndrewFarley]

The idea of gutting the Windows support from this module is quite appealing in some ways. It is some of the least covered code as far as testing goes, and that's because it's core developers and maintainers (or, whats left of us/them) don't use Windows at all. So it's no surprise to me that in the past randomly after releases were cut some issues popped up on Windows. I found it painful to navigate around the Windows support in this plugin while developing as well. I recall my larger PR took me months of extra work and rework to navigate the code to work properly on Windows, and only on windows, it worked fantastically on Linux/OS-X (as that's where I live/work/play).

Lets do this as a thought experiment @bsamuel-ui / @bryantbiggs what it would/might look like...

• To start things off we probably want to get some usage metrics from folks about what OS and what features they use of this plugin. We can either...
  • Publish a new version of this plugin that gathers anonymous information about which features are being used and on what OS, etc. Send it off to something free/easy/cheap (eg: google forms/sheets, or even AWS APIG/Lambda/DynamoDB which would still cost like nothing)
  • Or... add some voting into the issue-tracking to gather information and try to ask people (maybe add some printed stuff into a newly published version to ask people to go vote and tell us)

Both of these methods have flaws, some people will gripe about security/privacy if we do the former, and if we do the latter we'll likely have very biased and missing information. If we do gathering anonymous info, we could upon install and use we can print out and warn people right up front of what we're doing in order to help inform us of how best to support and maintain the plugin. People can opt-out simply with an ENV var, or a config option which we state very clearly also in the README and in the docs, and in the npm repo registry.

Moving forward from the data-gathering, depending on the information gathered we can do things such as...

• If Windows proves to be below a threshold (eg: 5 or 10%) then we can consider forking this repo to another named serverless-python-requirements-windows and do a call-out for maintainers of a windows fork. In that repo, they/we can strip Linux support from it completely and add CI testing focused on the various cases and conditions of windows and the various kinds of docker on windows. This would greatly simplify and streamline and shorten the development and testing cycles having a simpler codebase and less tests to run through and validate code from.
• This repository could then go through a major release removing Windows support entirely, removing the tests and the code for it and focusing on it working on unix-ey machines. This would similar as above, simplify the code, reduce the time to run tests, and allow us to focus on and deliver smaller/quicker iterations from focused functional tests.
• Based on the data-gathering after a few months months we could go through features below a threshold and consider/discuss removal.
• Finally, there's a interesting implementation nuances to this to discuss though. If we did fork/split, there's not an super easy way to conditionally install plugins in Serverless. I'm sure the general public would appreciate a way to ensure that a codebase could support Linux and Windows based development. There is some trickery in that serverless supports embedding Javascript. I've done in the past which would look something similar to this.

serverless.yaml:
custom:
  pythonRequirements: ${file(./build_helpers.js):get_python_reqs_config}
...

---

build_helpers.js:
var os = require("os")

// This gets the Python requirements configuration dependant on OS
module.exports.get_python_reqs_config = function() {
  if (os.platform() != 'linux')
    return Promise.resolve({invalidateCaches: false, useStaticCache: true, useDownloadCache: true, dockerizePip: true, dockerImage: "andrewfarley/serverless-personal-building-with-mysql"});
  return Promise.resolve({invalidateCaches: false});

I don't know how ideal, or comfortable that is. And I haven't tested if this embedding javascript works for plugins, would require some validation. Likely in the package.json you would have to install both, but only one gets used depending on what is in plugins:. If this method works, we could just have an working example demonstrating how it's still possible to allow devs any environment they want for development with a single codebase and single serverless.yaml config file.

How does any/all of that sound? Is it feasible to some of you/us?

I'd personally +1 the idea to remove Windows support and focus on Linux native and Docker-based support. Even without doing data-gathering, but that might be valuable as well to ensure we don't alienate a huge percentage of our userbase without being informed at least at first.

[software-2]

Windows user chiming in here to anecdotally show there's indeed much interest in continuing to use this project with Windows development environments. My teams have a mix of Linux/Mac/Windows users, and it would be a big downside if we had to worry about project fragmentation and hoping a fork is in sync with the main project, especially when the main project has threads questioning the life of the project.

[dschep]

Tagging @heri16 and @kichik as they're the contributors listed in the README for Windows support.

[bryantbiggs]

@software-2 I think that brings up the crux of all of this - the project is sort of in a weird cycle (if I understand correctly) at the moment. The core group (OG and new) that looks after this project, they do not work on Windows machines and therefore do not dabble in the Windows portion of the codebase. However, significant effort is required to make advancements/refactors/changes due to needing to support Windows. Its a tough spot, nobody is trying to screw anyone over, but at the same time, development on the project is stalling (presumably) because of this support for Windows. Personally it doesn't make sense to me - lambda environments are nix based (or nix-likeish), the containers used in this project to bundle/package the artifacts only work using *nix constructs (which is where WSL comes in), etc. Again, its a tough decision but it seems either a major release is cut where Windows support is dropped and the project is able to go ahead and flourish, or it stays on its current course which does not look promising unfortunately. If there is indeed a large Windows user base, perhaps they will come forward and we can figure out a game plan

[bsamuel-ui]

Publish a new version of this plugin that gathers anonymous information about which features are being used and on what OS, etc.

Now that the project is sponsored by Capital One, we have strict privacy requirements around gathering information. It's a good deal for our customers, but it makes that hard.

I checked some of the various npm download stats sites, none of them break out downloads per OS, so I'm guessing npm doesn't track that.

Windows is definitely contributing to making the tests hard to work with, but I'd like to see if there are easier ways to unblock this.

Is there any reason to maintain Python 2 support? If not, a next step is to spin off a new major version dropping support for that entirely.

[dschep]

Is there any reason to maintain Python 2 support? If not, a next step is to spin off a new major version dropping support for that entirely.

Agreed. Nuke Python 2. It's officially EOL, and AWS Lambda EOL is exactly 2 months from now according to this: https://aws.amazon.com/blogs/compute/continued-support-for-python-2-7-on-aws-lambda/

[bsamuel-ui]

Version 5.1.1 is now out, so let's mark this beautiful Norweigan blue has a bit of life left in him.