Some questions and Ideas

I found your question in the npx repo and found this repo, but I have a few questions.

Background

I'm working on various dev-utilities because many of the commonly used ones were designed around archaic JS/ES functionalities and are often incompatible with (or not worth the effort to make them work with) my es2020/esNext projects.

Currently my focus is around:

improved integration and simplified usage of package(-lifetime)-scripts and shell-like scripting

Package scripts are an incredible tool for both development and in deployment but setting them up is complex and time consuming and creating a good default implementation is even harder. so I've been experimenting around and I think using small and standardized modules that can be scripted and used programmatically as needed is the smartest way to go.

One thing I think this could enable are small scripts and services able to work together and independently, sharing data in a loose swarm where both shell-like scripting and programmatic usages can work together to be both flexible and not configuration intensive. this would be the holy grail, but this will be incredibly hard to make functional.

simplifying dependency maintenance and dep-tree integrity to avoid Dependency-Hell

Many parts of the JS ecosystem are stuck in a dependency-hell, because they were started when invariance of packages was the common way to ensure stability in programming. The language and its foundations have been advancing at high speed for many years now, yet this principle of invariance is still common and techniques that have been considered bad practice for years or have much much simpler modern alternatives are still regularly recommended all over the place.

It would be great for new projects to be as easy as possible to modernise as they age. I think the smart way to do that is focusing on easy adaptability instead of base-level compatibility and on creating flexible interfaces that can be switched out easily if needed and basing convention at this top-layer instead. Pressuring users to update/-grade constantly by issuing deprecation early, and making it easy to do so is the smart move now. for this it's important to look at the upcoming changes in the language and your frameworks and prepare ahead of time to adapt as soon as possible so the user has the ability to do the same.

general maintenance automation based on and around a modern infrastructure

The physical infrastructure has advanced to a point where resources for programming is abundant. Bandwidth is large enough to support updating something for even the most minor changes, and processing power is plenty enough to do so passively with negligible performance impact instead of on-use.

For this I plan on working on a way to include micro-services seamlessly in a project with similar functionalities as GitHub-actions but for client- and dev-side use instead of in a repository

easier creation and configuration of projects implementing and aiding in creation of the aforementioned points

This is probably the biggest roadblock for me right now. Having a boiler-plate be as current as possible will require adaptive automation, dynamic recommendations and an easier way to configure and initialize complex projects than anything I've found. For this reason I think my first point is the most urgent.

following

I see you are working on a few similar things and I'd be very interested in exchanging ideas.

on that note, I see that your code base is rather modern. What are your thougts on switching over to using ES-modules? They would solve a couple of the issues you talk about in the readme not with anything that adds complexity but by their very nature. eg.

named import/export can take care of many issues with function naming conventions, as they only need to have consistency at the surface level.
The nature of ESM allows static analyzing across modules and import/export can be automatically and correctly typed by the typescript-language-feature for intellisense and .d.ts-files making very strict convention less relevant.
it is framework independent, stripping one layer of complexity and abstraction.
very little extra code is needed to make a scripts behave consistently on the command line and when used programmatically.
usage of top-level-await: it's still experimental in v8 but it's part of the EcmaScript-standard and it will be implemented consistently on all platforms in the near future. (can be enabled via --harmony-top-level-await)

on a related note, I see that you added typings, you could switch to typescript directly, as with modern ES-features and using vanilla-style typescript and ESM, you can write code that only needs to be stripped of type-annotations to be valid JS/ES-code, this way the .d.ts-files don't have to be written seperatly and the code gets basic documentation as you go, making coding more consistent and modifying code easier and less error-prone.

If you need a more detailed explanation on any of this, feel free to just ask.

and my apologies for writing that much, lol

I must admit at some point I scrolled to the bottom of your post, trying to find your questions.

I'm more familiar with typescript then with JS module building. But I know enough about it, to be able to build this tool, as of now.

Using less tools and writing code against such a low level API (currently) brings the benefit of reduced complexity and things to consider. But I already hit some walls when trying refactor this package to be more modular, so it might make sense to switch to more sophisticated tooling.

But: One goal/decision was to be able to work with node 10. So it's nice that with newer versions this tool would need less code, but as long as node v10 is a requirement for me, this could only be an optimization. Does this "limitation" change anything towards your statements above? Would it be possible to generate the code for older node versions from the things you are proposing?

Currently the module is doing mostly great for what I need it to do, and although I have some ideas, it's not currently in my focus. (Contributions, suggestions, ideas, questions are very welcome and I'm open for PRs!)

Let me know if I addressed/answered at least some of your questions. And if not, let's talk about them one by one: pick one, and try to be as precise as possible when putting it in a reply. (I would love to understand and have an answer to all you topics/questions, but currently they are very broad for me and understanding all details and implications could take me days...)

I must admit at some point I scrolled to the bottom of your post, trying to find your questions.

yeah, it turned out to be less about asking questions as about introducing ideas...

(Contributions, suggestions, ideas, questions are very welcome and I'm open for PRs!)

It's also not about nescessarily transforming this project. (even thogh it has a great name. I can't believe that it was still awailable on npm)

(I would love to understand and have an answer to all you topics/questions, but currently they are very broad for me and understanding all details and implications could take me days...)

sure. it wasn't suposed to be ultra clear anyways, it's for just for starting a discord, ordering my thougts and to serve as notes. so no worries.

before i go into detail:

I'm more familiar with typescript then with JS module building. But I know enough about it, to be able to build this tool, as of now.

I took a look at one of your TS projects, the import/export you use there is modeled after EcmaScript-module-specs, so you are allready plenty familiar with writing es-modules. the only real difference between how typescript does it and vanilla-JS/ES is that you need to add a file extention if you are directly importing a file. eg:

import * from './path/to/file'
// becomes:
import * from './path/to/file.js'

// packages can be imported the same way tho
import * from 'some-package'

and if the source is written in typescript, i think that will be handled by the compiler. i always include the extension in typescript too, as it's closer to how it's done in JS

So this paragrapf from my post:

on a related note, i see that you added typings, you could switch to typescript directly, as with modern ES-features and using vanilla-style typescript and esm, you can write code that only needs to be stripped of type-annotations to be valid JS/ES-code, this way the .d.ts-files don't have to be written separatly and the code gets basic documentation as you go, making coding more consistent and modifying code easier and less error-prone.

becomes moot, the answer is: in your ts-projects, You are allready doing exactly what i suggested. So this probably answers many of your questions: you should be able to to all the things you are able to do in your ts-projects

for reference, i plopped a file from tslint-report into the ts-playground with the target and module options set to ESNEXT. follow the link, there you can see it side by side.

But: One goal/decision was to be able to work with node 10. Would it be possible to generate the code for older node versions from the things you are proposing?

It can be transpiled to work with node 10 without a problem. And to directly run a ts file ts-node has been working on experimental esm-support for a while. runtime compilation of ts should't be used in production anyways, but it's great for testing and developing.

Does this "limitation" change anything towards your statements above?

Implementation of the things i mentioned is certainly possible in earlier versions and without using esm, but:

it would make the goal of adapting easily and fast to the newest developments considerably harder.
it would be pretty ironic if the goals are perfectly achieved with methods counter to the reasons for implementing them
I'm utterly incapable to write code for those versions, as i learend the language using all the modern convenience. it would be like using a pocket-knife to cut down a tree instead of the chainsaw i'm used to.

I do have an actual question this time:

But: One goal/decision was to be able to work with node 10.

may i ask why? just curious

*Note: node v10 reaches end-of-life in less than a year (2021-04-30), that's not too far away, don't forget that. And v14 will enter active long-term-support soon (2020-10-20), officially labeling esm as stable.

I hope things have now become a lot clearer

The very personal answer is that there is a piece of infrastructure this script is used on that can currently not be updated. The more general answer is that for libraries and tools it is very common to only stop supporting a node version (some month) after EOL, as could be observed with node v8 this year, the had end of life in December 19. Then the next step would be v12 which I would at least want to support until end of 2022.

Relying on ts-node for execution of this lib is not an option for me:

it adds one more dependency
I have made very bad experience with relying on it
it adds a level of complexity that I don't want in this very thin layer
I assume it would have a performance impact, but that's just a guess Note: it's of course possible to use this lib with ts-node/ts modules already, as the tests show.

Thx for elaborating on the differences between ES modules and typescript. As soon as I found a solid way to transpile that to something that works with node v10 it could be applied. But I will need to do some research here and I'm not sure how soon I'll be able to pit time into it. Since the code base is still very small, if you have an idea how to do that, PRs are welcome.

from my last comment:

runtime compilation of ts should't be used in production anyways, but it's great for testing and developing.

I'm not that mad, lol

and asfar as compiling goes, it's very straight forward. It's just 2 settings in the the tsconfig.json. the compiler will do the rest for you. same file as before on the ts-playground but with the options {module: 'commonjs', target: es2017}', i can make a sample PR showing it.

But again, it's not my goal to neccessarily transform this project. I must admit, i went a bit over the rails with my 'introduction'. What brought me here is you expressing interest possibly working on npx, and since it's not looking very bright i'm doing some scouting for people who might be interested in working on an alternative tool or similar projects

But again, it's not my goal to necessarily transform this project. I must admit, i went a bit over the rails with my 'introduction'. What brought me here is you expressing interest possibly working on npx, and since it's not looking very bright i'm doing some scouting for people who might be interested in working on an alternative tool or similar projects

Ah although it was the first thing you wrote initially, all the text after it made me not pay attention.

So I went back and read your initial post again. (And it's about time to say: I now finally understood that your main attention is not this library. I guess I didn't pay attention since you created this issue in this repo.)

So: I agree that the current status of npx is at least a pitty, but rather unbearable. I'm open for discussing ideas for something that could eventually replace is, or even provide something that is even easier to adopt and maintain.

But even after (finally) reading your whole text, I don't understand anything actionable or answerable in it.

So what exactly do you want to discuss? Any repo or whiteboard I should have a closer look at? (I quickly looked over your github activity, but there was to much for something abvious to stick out.) And maybe there is a more efficient way of doing that then comments in this issue?

Best Christian

And it's about time to say: I now finally understood that your main attention is not this library.

Yeah, completely my fault. As i said, i went overboard. I was trying to extensively express my ideas and kinda forgott about the framing.

But even after (finally) reading your whole text, I don't understand anything actionable or answerable in it.

no worries. it's currently mostly a thoughtmap. it's important that i understand it's clear meaning so i don't forget about stuff

Any repo or whiteboard I should have a closer look at?

Not currently, I'm technically still in the idea-phase. this thread was never really supposed to be what it's become. But for such things GH-issues are great because they create a nice creation-process documentation. Can't really make a specific project because these ideas can't be just a single project. But i think it's about time to create a general-scope project for discussing these ideas.

Again, my apologies for abusing your project for my rambeling. I'll tag you on the project once it's up. feel free to continue this thread for questions about the ts/js transpiling vor version compatability.

EDIT: and don't judge my skill or conventions by my repos. most of them are a comlpete mess and were created mostly to experiment, lol

OK, sure, count me in on brainstorming, experiments, "whatever" :wink:

karfau / runex