Open steelbrain opened 8 years ago
This is good feedback - thank you. These are definitely edge cases, as the current deployment strategy works for the majority of the thousands of packages on atom.io, but as we want to embrace more comprehensive IDE features helping people keep their repos clean would be helpful.
As you note, backwards-compatibility is important. Another concern of mine is that releases be consistent - that is to say, we want to avoid someone shipping a release and then changing the downloaded archive later - version 1.1 (or whatever) should always be the exact same code.
This probably won't make it onto our roadmap in the very near future, but I think it's interesting to have a discussion here about what we can do to address the various pain points - the one most interesting to me is transpilation - are there other ways this could work to prevent check-in of build products without adding a second place to put products for download? e.g. a transpilation step on the server side for the most popular targets like typescript, etc?
e.g. a transpilation step on the server side for the most popular targets like typescript, etc?
I think Atom should generalize its involvement in transpilers rather than have first class support for a few (looking at you, CoffeeScript and Babel), and enable developers to set up easily with whatever transpiler they choose. VSCode has generalized transpilers/compilers by hooking them into the package's config and into VSCode's task runner for refresh/reload development. While all the transpiler examples talk about and show TypeScript, they demonstrate they are just shelling out to the TS Compiler, tsc
. I think it'd be great if this lead to Atom bundling no transpilers by default.
Ideally developers shouldn't have to commit transpiled code in order to release it, and they should be able to exclude the source files from the released package. An '.apmignore' file would accomplish the second part, like @steelbrain mentioned, and that is the direction that VSCode went with its '.vscodeignore'. Their Yeoman project generators are set up for TypeScript, but VSCode only runs JavaScript. The projects add **/*.ts
to their '.vscodeigore' files on generation.
I recently learned it's possible to exclude resources from package distributions by using .gitattributes
:
/docs/ export-ignore
/1.5mbs-animated-banner-for-your-readme.gif export-ignore
/other-rubbish-that-shreds-bandwidth/ export-ignore
This attribute relates to the git archive
command, which is clearly responsible for generating the tarballs GitHub serves to those downloading zipped repos. Examining the output of apm install --verbose
revealed this is how packages are being downloaded:
via: '1.1 vegur' }
REQUEST end event https://atom.io/api/packages/jss-atom-snippets
REQUEST has body https://atom.io/api/packages/jss-atom-snippets 3034
REQUEST emitting complete https://atom.io/api/packages/jss-atom-snippets
REQUEST { url: 'https://www.atom.io/api/packages/jss-atom-snippets/versions/1.3.0/tarball',
strictSSL: true,
headers: { 'User-Agent': 'npm/3.10.5 node/v4.4.5 darwin x64' },
method: 'GET',
callback: undefined }
REQUEST make request https://www.atom.io/api/packages/jss-atom-snippets/versions/1.3.0/tarball
REQUEST onRequestResponse https://www.atom.io/api/packages/jss-atom-snippets/versions/1.3.0/tarball 302 { server: 'Cowboy',
connection: 'close',
date: 'Sun, 13 Nov 2016 00:59:29 GMT',
status: '302 Found',
'x-frame-options': 'DENY',
'x-xss-protection': '1; mode=block',
'x-content-type-options': 'nosniff',
'strict-transport-security': 'max-age=631152000',
'content-security-policy': 'default-src \'self\'; connect-src \'self\'; font-src https://github-atom-io-herokuapp-com.global.ssl.fastly.net; frame-src \'self\' https://www.youtube.com; img-src https://* \'self\' https://github-atom-io-herokuapp-com.global.ssl.fastly.net data:; media-src \'none\'; object-src \'self\' https://github-atom-io-herokuapp-com.global.ssl.fastly.net; script-src \'self\' \'unsafe-inline\' https://ssl.google-analytics.com https://www.google-analytics.com https://platform.twitter.com https://github-atom-io-herokuapp-com.global.ssl.fastly.net; style-src \'self\' \'unsafe-inline\' https://github-atom-io-herokuapp-com.global.ssl.fastly.net;',
location: 'https://codeload.github.com/march213/jss-atom-snippets/legacy.tar.gz/v1.3.0',
'content-type': 'text/html; charset=utf-8',
'cache-control': 'no-cache',
'x-request-id': 'e867ee1d-4778-469f-a255-8925c8c5f385',
'x-runtime': '0.063747',
'x-rack-cache': 'miss',
vary: 'Origin',
via: '1.1 vegur' }
REQUEST redirect https://codeload.github.com/march213/jss-atom-snippets/legacy.tar.gz/v1.3.0
REQUEST redirect to https://codeload.github.com/march213/jss-atom-snippets/legacy.tar.gz/v1.3.0
This seems to be a more logical approach than using an .apmignore
or .npmignore
file... and the fact that it's drawn from Git itself makes it feel like a "canonical" answer to the issue of auxiliary resource exclusion.
However, the fact this works based solely on a redirect to GitHub makes me hesitant to embrace it as a long-term solution. How likely is it that Atom's package repository will continue serving tarballs this way? Infrastructure changes are inevitable, and packages might one day be served using a strategy that doesn't involve a call to git archive
.
The existence of export-ignore
makes me wonder why NPM feels .npmignore
files are necessary... the world needs more vendor-specific manifests like it needs another Vietnam War.
For myself, I have the opposite problem @alhadis. There are files which I want excluded from git (because they are build artifacts, and so diffs etc aren't useful) but included in my atom plugin. .npmignore allows this because npm ignores the .gitignore file when it is present.
Diffs can easily be suppressed using .gitattributes
too, although not on GitHub (which is a known issue; just check Linguist's backlog). Still, you're right... though I don't understand why NPM requires ignored paths to be specified in an external file, rather than package.json
.
This is good feedback - thank you. These are definitely edge cases, as the current deployment strategy works for the majority of the thousands of packages on atom.io, but as we want to embrace more comprehensive IDE features helping people keep their repos clean would be helpful.
Is this still considered an edge case? The JS ecosystem has changed since 2016, now almost everybody uses Babel, and Typescript is getting a lot of traction. It feels bad to have the transpiled JS in the repo just to include it in the package.
Just to give some insights on modern package development, I would like to share my workflow, hoping that .apmignore
will be supported in the future.
For my own package development, I'm following Microsoft's suggestion for VSCode to use bundling.
Visual Studio Code extensions often grow quickly in size. They are authored in multiple source files and depend on modules from npm. Decomposition and reuse are development best practices but they come at a cost when installing and running extensions. Loading 100 small files is much slower than loading one large file. That's why we recommend bundling. Bundling is the process of combining multiple small source files into a single file.
In practice, I'm using Webpack to create a bundle from all files in /src
– no matter if my packages are written in JavaScript, CoffeeScript and TypeScript. While this approach gives me excellent performance, it comes with redundancy. The end-user neither needs the /src
folder nor node_modules
. To get rid off the latter, I guess I could declare all dependencies
as devDependencies
, but that doesn't feel right.
Ultimately, I think that if many package developers adopt bundling strategies, not only their packages will benefit, but Atom as a whole. Calling this feature request an edge-case is a bit short-sighted, don't you think?
I made a solution to this. We can use build-commit:
dist
) to .gitgnore
scripts
entry of package.json
"build-commit": "build-commit -o dist",
npm run build-commit
before doing apm publish
.To get rid off the latter, I guess I could declare all
dependencies
asdevDependencies
, but that doesn't feel right.
@idleberg I think it is right, though for your case.
For the record, if you are shipping a transpiled copy of all of your required modules or functions, and loading them from the transpiled output at runtime, then end-users don't need your dependencies
anymore, and they genuinely are development-only dependencies, when referring to their non-transpiled, module form.
Specifying a dependency
is requesting that your package manager "please download these modules during the install of my package, thanks." The devDependencies
field seems totally correct for your use-case.
Summarizing what has been said earlier in the thread: some of these capabilities folks are asking for here are already available.
export-ignore
in .gitattributes
. https://github.com/atom/apm/issues/498#issuecomment-260161290npm
package registry and depend on that package.That leaves some room for improvement, so yeah, I agree being able to actually directly upload tarballs of Atom packages would be an improvement. I just thought it would be nice to restate that info simply and put it in one place, in case folks are in need of a workaround today.
While being able to upload tarballs would be an improvement in capabilities, there are some downsides as well:
npm
-style package registry. Which is not unheard-of, and the npm
tooling is all meant to work with arbitrary package registries, so this isn't a "rewrite npm" problem, it's strictly the trouble of setting up and supporting the infrastructure.
npm
package registry, the uploader can include arbitrary files. But the npm
ecosystem already has to deal with this problem, and so does the Atom package ecosystem, since npm
packages are already allowed as dependencies.
The way releases are handled at the moment are great, but it has a lot of issues and problems
The problems
.apmignore
file in the repoThe solution Create tar archive (or zip if you prefer) respecting ignore files and counting in files ignored by git repo from the the local version of the publisher. Add a release with the newly created tag and attach that archive with that release.
The release tag should be prefixed by package name, like
linter-ruby-v1.1
. The archive name attached to the release should be namedpackage-name.tgz
orpackage-name.zip
.When installing a package, check for the latest tag in APM database for that package, if the tag starts with
package-name
, try to find release for that tag and download the package file. Otherwise, just download the zip archive of commit as we do now. If we follow this pattern, we would be backward-compatible with the already released packages.I know the problems I mentioned above do not sound very important, but they are very annoying and very difficult to workaround.
For example @basarat, @blakeembrey and the rest of the
atom-typescript
team transpiles the typescript files and keeps them in their repo.@nmote @ssorallen @bolinfest and the rest of the nuclide team had to create a new github org
facebooknuclideapm
and create a repo in it for each package they wanted to publish.@david-driscoll and the rest of the
omnisharp-atom
team also has to transpile typescript files and keep them in the repo.@devoncarew and the rest of
dart-lang
team has to keep transpileddart
files in their repo.@steelbrain @Arcanemagus and the rest of
linter-eslint
team has to commit transpiled babel files into the repo, because we spawn a child process that requires our files and child processes don't support babel requires