.npmrc is not matched by gulp.src

localjo commented 8 years ago

I'm trying to match all dotfiles with gulp.src. I've tried several options. I'm 95% sure these two should work to match all files, including dotfiles; gulp.src('/path/to/my/files/**/{*,.*}') gulp.src('/path/to/my/files/**/*', {dot: true}) Both patterns match all non-dotfiles, and some dotfiles, but some dotfiles are missing. Specifically, I've noticed that .npmrc files are missing. When my CWD contains an .npmrc, I see this output when I use gulp-debug to inspect the stream;

[12:10:56] file: /path/to/my/files/README.md
[12:10:56] file: /path/to/my/files/assets
[12:10:56] file: /path/to/my/files/auth.json
[12:10:56] file: /path/to/my/files/gulpfile.js
[12:10:56] file: /path/to/my/files/metadata.yml
[12:10:56] file: /path/to/my/files/package.json
[12:10:56] file: /path/to/my/files/views
[12:10:56] file: /path/to/my/files/.editorconfig
[12:10:56] file: /path/to/my/files/.npmignore
[12:10:56] file: /path/to/my/files/assets/fonts
[12:10:56] file: /path/to/my/files/assets/images
[12:10:56] file: /path/to/my/files/assets/scripts
[12:10:56] file: /path/to/my/files/assets/styles
[12:10:56] file: /path/to/my/files/assets/.gitkeep
[12:10:56] file: /path/to/my/files/views/index.hbs
[12:10:56] file: /path/to/my/files/views/layout.hbs
[12:10:56] file: /path/to/my/files/views/.gitkeep
[12:10:56] file: /path/to/my/files/assets/fonts/.gitkeep
[12:10:56] file: /path/to/my/files/assets/images/favicon.ico
[12:10:56] file: /path/to/my/files/assets/images/.gitkeep
[12:10:56] file: /path/to/my/files/assets/scripts/index.js
[12:10:56] file: /path/to/my/files/assets/scripts/.gitkeep
[12:10:56] file: /path/to/my/files/assets/styles/index.scss
[12:10:56] file: /path/to/my/files/assets/styles/.gitkeep

Here's my full gulp task, for reference;

gulp.task('default', function(done) {
  gulp.src(/path/to/my/files/**/{*,.*}')
    .pipe(debug({title: 'file:'}))
    .pipe(template(options))
    .pipe(gulp.dest('./' + options.siteSlug));
});

I've also checked my .gitignore and .npmignore files to make sure that there are no entries that would exclude an .npmrc file, although I don't know whether that would matter. I'm using npm v3.3.6 and node v5.0.0 Anyway, is there any particular reason that some dotfiles would be excluded in this case? Can you reproduce this issue?

localjo commented 8 years ago

Even gulp.src('/path/to/my/files/.npmrc') doesn't seem to match the .npmrc file.

localjo commented 8 years ago

For reference, ls -lhA in /path/to/my/files returns;

-rw-r--r--  1 Josiah  staff   114B Feb  3 16:49 .editorconfig
-rw-r--r--  1 Josiah  staff   147B Feb  3 16:49 .gitignore
-rw-r--r--  1 Josiah  staff    46B Feb  3 16:49 .npmrc
-rw-r--r--  1 Josiah  staff   729B Feb  3 16:49 README.md
drwxrwxr-x  8 Josiah  staff   272B Feb  3 16:49 assets/
-rw-r--r--  1 Josiah  staff   507B Feb  3 16:49 auth.json
-rw-r--r--  1 Josiah  staff   1.5K Feb  3 16:49 gulpfile.js
-rw-r--r--  1 Josiah  staff   249B Feb  3 16:49 metadata.yml
-rw-r--r--  1 Josiah  staff   620B Feb 10 10:24 package.json
drwxr-xr-x  5 Josiah  staff   170B Feb 10 10:24 views/

localjo commented 8 years ago

And gulp.src('/path/to/my/files/.editorconfig') does match the .editorconfig file.

localjo commented 8 years ago

It may be relevant that the contents of the .npmrc file I'm using are;

//registry.npmjs.org/:_authToken=${NPM_TOKEN}

localjo commented 8 years ago

If I rename the .npmrc file, the ${NPM_TOKEN} breaks the stream with an error. But if I remove ${NPM_TOKEN} from .npmrc, the file still isn't picked up by gulp.src. So it seems like there are compounded issues at play here. The broken stream with a variable is easy enough to work around, so I'm going to focus on figuring out why an .npmrc file with any contents is not picked up.

localjo commented 8 years ago

This bit from the npm docs might be relevant:

NOTE: Because local (per-project or per-user) .npmrc files can contain sensitive credentials, they must be readable and writable only by your user account (i.e. must have a mode of 0600), otherwise they will be ignored by npm!

localjo commented 8 years ago

I'm not sure how that plays out in the grand scheme of how everything is executed.

tunnckoCore commented 8 years ago

It seems it is node-glob related issue.

'use strict'

var glob = require('glob')

glob('./*.*', {dot: true}, function (err, res) {
  console.log(res)
})

output

[ './.codeclimate.yml',
  './.editorconfig',
  './.git',
  './.gitignore',
  './.npmrc',
  './.travis.yml',
  './CHANGELOG.md',
  './CONTRIBUTING.md',
  './index.js',
  './package.json',
  './rc.js',
  './README.md',
  './test.js' ]

localjo commented 8 years ago

@tunnckoCore I'm not sure if I understand what you're saying. That code doesn't seem to be related to .npmrc, unless I'm missing something. Other dotfiles are working fine.

tunnckoCore commented 8 years ago

~~Gulp relies on node-glob library and simple test as above shows that even just glob nto match the npmrc but the all of other dotfiles. (i'll update the example, i missed something)~~

localjo commented 8 years ago

Oh, gotcha. Ok, I'll open an issue over there.

tunnckoCore commented 8 years ago

Hah, wait, i'm wrong. Sorry.

tunnckoCore commented 8 years ago

Which gulp version you use? 3.9 or latest a.k.a. v4 ? Try with above pattern ./*.*

localjo commented 8 years ago

I'm actually using this in slush, which uses ~3.6.2 https://github.com/slushjs/slush/blob/master/package.json#L39

tunnckoCore commented 8 years ago

i think this is very bad, but it's their and yours decision. Please try raw glup 3.9, or latest glob-stream like that

'use strict'

var glob = require('glob-stream')

var stream = glob.create(['./*.*'], {dot: true})
stream.on('data', console.log)

I think it is some old issue, because all works for me with latest versions, but i don't know exactly.

phated commented 8 years ago

Yep. Please upgrade gulp to 3.9.1

localjo commented 8 years ago

Hmm, I'll be digging into this more tomorrow, but I updated slush to use ^3.9.1 and my glob still isn't picking up .npmrc.

localjo commented 8 years ago

This picks up .npmrc;

'use strict'

var glob = require('glob-stream')

var stream = glob.create(__dirname + '/my/files/**/{*,.*}')
stream.on('data', console.log)

localjo commented 8 years ago

I'm not picking up .npmrc in my slushfile with the following installed globally;

$ npm ls --global --depth 1
/Users/Josiah/.nvm/versions/node/v5.0.0/lib
├─┬ gulp@3.9.1
│ ├── archy@1.0.0
│ ├── chalk@1.1.1
│ ├── deprecated@0.0.1
│ ├── gulp-util@3.0.7
│ ├── interpret@1.0.0
│ ├── liftoff@2.2.0
│ ├── minimist@1.2.0
│ ├── orchestrator@0.3.7
│ ├── pretty-hrtime@1.0.1
│ ├── semver@4.3.6
│ ├── tildify@1.1.2
│ ├── v8flags@2.0.11
│ └── vinyl-fs@0.3.14
├─┬ npm@3.7.1
│ ├── abbrev@1.0.7
│ ├── ansi-regex@2.0.0
│ ├── ansicolors@0.3.2
│ ├── ansistyles@0.1.3
│ ├── aproba@1.0.1
│ ├── archy@1.0.0
│ ├── async-some@1.0.2
│ ├── chownr@1.0.1
│ ├── cmd-shim@2.0.1
│ ├── columnify@1.5.4
│ ├── config-chain@1.1.10
│ ├── debuglog@1.0.1
│ ├── dezalgo@1.0.3
│ ├── editor@1.0.0
│ ├── fs-vacuum@1.2.7
│ ├── fs-write-stream-atomic@1.0.8
│ ├── fstream@1.0.8
│ ├── fstream-npm@1.0.7
│ ├── glob@6.0.4
│ ├── graceful-fs@4.1.2
│ ├── has-unicode@2.0.0
│ ├── hosted-git-info@2.1.4
│ ├── iferr@0.1.5
│ ├── imurmurhash@0.1.4
│ ├── inflight@1.0.4
│ ├── inherits@2.0.1
│ ├── ini@1.3.4
│ ├── init-package-json@1.9.3
│ ├── lockfile@1.0.1
│ ├── lodash._baseindexof@3.1.0
│ ├── lodash._baseuniq@3.0.3
│ ├── lodash._bindcallback@3.0.1
│ ├── lodash._cacheindexof@3.0.2
│ ├── lodash._createcache@3.1.2
│ ├── lodash._getnative@3.9.1
│ ├── lodash.clonedeep@4.0.2
│ ├── lodash.isarguments@3.0.5
│ ├── lodash.isarray@4.0.0
│ ├── lodash.keys@4.0.0
│ ├── lodash.restparam@3.6.1
│ ├── lodash.union@4.0.1
│ ├── lodash.uniq@4.0.1
│ ├── lodash.without@4.0.1
│ ├── mkdirp@0.5.1
│ ├── node-gyp@3.2.1
│ ├── nopt@3.0.6
│ ├── normalize-git-url@3.0.1
│ ├── normalize-package-data@2.3.5
│ ├── npm-cache-filename@1.0.2
│ ├── npm-install-checks@3.0.0
│ ├── npm-package-arg@4.1.0
│ ├── npm-registry-client@7.0.9
│ ├── npm-user-validate@0.1.2
│ ├── npmlog@2.0.2
│ ├── once@1.3.3
│ ├── opener@1.4.1
│ ├── osenv@0.1.3
│ ├── path-is-inside@1.0.1
│ ├── read@1.0.7
│ ├── read-cmd-shim@1.0.1
│ ├── read-installed@4.0.3
│ ├── read-package-json@2.0.3
│ ├── read-package-tree@5.1.2
│ ├── readable-stream@2.0.5
│ ├── readdir-scoped-modules@1.0.2
│ ├── realize-package-specifier@3.0.1
│ ├── request@2.69.0
│ ├── retry@0.9.0
│ ├── rimraf@2.5.1
│ ├── semver@5.1.0
│ ├── sha@2.0.1
│ ├── slide@1.1.6
│ ├── sorted-object@1.0.0
│ ├── strip-ansi@3.0.0
│ ├── tar@2.2.1
│ ├── text-table@0.2.0
│ ├── uid-number@0.0.6
│ ├── umask@1.1.0
│ ├── unique-filename@1.1.0
│ ├── unpipe@1.0.0
│ ├── validate-npm-package-license@3.0.1
│ ├── validate-npm-package-name@2.2.2
│ ├── which@1.2.4
│ ├── wrappy@1.0.1
│ └── write-file-atomic@1.1.4
├─┬ slush@1.1.1
│ ├── archy@0.0.2
│ ├── chalk@0.4.0
│ ├── glob@7.0.0
│ ├── gulp-util@2.2.20
│ ├── liftoff@0.10.0
│ ├── minimist@0.1.0
│ └── pretty-hrtime@0.2.2
└─┬ slush-solidus@1.0.0
  ├── glob-stream@5.3.1
  ├── gulp@3.9.1
  ├── gulp-debug@2.1.2
  ├── gulp-template@3.1.0
  ├── inquirer@0.8.5
  ├── minimist@1.2.0
  └── path@0.11.14

Here is my slushfile.js

var gulp = require('gulp');
var template = require('gulp-template');
var debug = require('gulp-debug');

var options = {....};

gulp.task('default', function(done) {
  gulp.src(__dirname + '/templates/site/**/{*,.*}')
    .pipe(debug())
    .pipe(template(options))
    .pipe(gulp.dest('./' + options.siteSlug));
});

Is there any way to log which version of gulp/glob/minimatch, etc is actually being used by my slushfile, so that I can track down where .npmrc might be excluded?

tunnckoCore commented 8 years ago

Strange.

Is there any way to log which version of gulp/glob/minimatch, etc is actually being used by my slushfile

it's a slush question, ask them what they do or look directly at their code. From deps tree, i'm seeing that they have minimatch and glob, so they do something. I'm almost sure, that the problem is from slush.

It looks pretty out-dated and pretty ugly. Try some of the upcoming awesome @jonschlinkert projects generate, scaffold, templates, base, or assemble-core and etc. They are built on top of gulp tools and are pretty stable and up to date.

Otherwise, open issue on slush and hope they answer, lol.

Cheers, Charlike.

phated commented 8 years ago

@josiahsprague I'm not sure if slush picks up gulp from the global (usually npm modules don't resolve that way). Are you required to have a local gulp install to use slush?

localjo commented 8 years ago

Turns out that my problem was that I was installing my module before running the code, and npm install didn't carry over the .npmrc file from my original project directory into the installed directory where my code was actually running. 🙈 Sorry everyone.

phated commented 8 years ago

Glad you got to the bottom of it and thanks for following up!

Krispomoho commented 7 years ago

I have met the same problem , I just want use gulp to move directories, but it will miss the file like .gitignore or .babelrc

here is my gulp code gulp.src('source/**/*').pipe(gulp.dest('./dest/'))

callumacrae commented 7 years ago

@Krispomoho: Has been answered in your other issue: https://github.com/gulpjs/gulp/issues/1849

gulpjs / gulp

.npmrc is not matched by gulp.src #1523