Directory.listSync() throws exception for an existing directory

[scheglov]

import 'dart:io';
main() {
  String path =
      'C:\\dart_projects\\angular\\node_modules\\systemjs\\node_modules\\es6-module-loader\\node_modules\\grunt-contrib-uglify\\node_modules\\maxmin\\node_modules\\pretty-bytes\\node_modules\\meow\\node_modules\\indent-string\\node_modules\\repeating\\node_modules\\is-finite\\node_modules';
  print(path.length);
  var directory = new Directory(path);
  print(directory.existsSync());
  directory.listSync();
}

The directory "path" does exist, and the file system manager that I use works just fine with it.

But directory.existsSync() says false. Why?!

And then directory.listSync() throws an exception.

Observatory listening on http://127.0.0.1:50917
261
false
Breaking on exception: object of type FileSystemException: FileSystemException: Directory listing failed, path = 'C:\dart_projects\angular\node_modules\systemjs\node_modules\es6-module-loader\node_modules\grunt-contrib-uglify\node_modules\maxmin\node_modules\pretty-bytes\node_modules\meow\node_modules\indent-string\node_modules\repeating\node_modules\is-finite\node_modules\*' (OS Error: Системе не удается найти указанный путь.
, errno = 3)
Unhandled exception:
FileSystemException: Directory listing failed, path = 'C:\dart_projects\angular\node_modules\systemjs\node_modules\es6-module-loader\node_modules\grunt-contrib-uglify\node_modules\maxmin\node_modules\pretty-bytes\node_modules\meow\node_modules\indent-string\node_modules\repeating\node_modules\is-finite\node_modules\*' (OS Error: Системе не удается найти указанный путь.

, errno = 3)
#0      _Directory._list (dart:io-patch/directory_patch.dart:17)
#1      _Directory.listSync (dart:io/directory_impl.dart:229)
#2      main (file:///C:/Users/Kosta/dart/test/bin/test.dart:11:13)
#3      _startIsolate.<anonymous closure> (dart:isolate-patch/isolate_patch.dart:261)
#4      _RawReceivePortImpl._handleMessage (dart:isolate-patch/isolate_patch.dart:148)

[sethladd]

@scheglov what is the impact of this bug?

[lrhn]

An immediate guess: The path is 261 chars long, and it's on Windows, so we probably hit the MAX_PATH == 260 limit on path lengths.

[alexander-doroshko]

@sethladd @lrhn The impact is that Dart Analysis Server crashes on projects that have long file paths. Originally we've got this bug report from a WebStorm user. And after that we reproduced the issue ourselves. Project to reproduce is https://github.com/angular/angular. After you run npm install in it you get long paths in the project (for node_modules) like the one in the issue description. All Analysis Server based IDEs will fail to work with such projects.

[scheglov]

@lrhn The guess is probably right, but why only Dart has this problem with long paths? I can get list of files of this directory just fine even in Java.

[nex3]

It looks like this error may also be top-leveled without passing through the stream for Directory.list. Konstantin sent me this example, using Directory.list within the watcher package:

main() {
  Chain.capture(() {
    String path =
        'C:\\dart_projects\\angular\\node_modules\\systemjs\\node_modules\\es6-module-loader\\node_modules\\grunt-contrib-uglify\\node_modules\\maxmin\\node_modules\\pretty-bytes\\node_modules\\meow\\node_modules\\indent-string\\node_modules\\repeating\\node_modules\\is-finite\\node_modules';
    String path2 =
        'C:\\dart_projects\\angular\\node_modules';
    print(path.length);
    var directory = new Directory(path);
    print(directory.existsSync());
    new DirectoryWatcher(path2).events.listen((_) => print(_));
  });
}

This produces the following output. Note in particular that the last time watcher appears in the stack trace is this line of directory_watcher/windows.dart, and on that line, we're setting up an onError handler for the stream. The error handler then forwards the error to the watcher's stream, which Konstantin's code is listening to. And a stream shouldn't top-level errors; this makes me think that the error isn't reaching the Directory.list stream at all.

Observatory listening on http://127.0.0.1:51338
261
false
Unhandled exception:
Uncaught Error: FileSystemException: Directory listing failed, path = 'C:\dart_projects\angular\node_modules\systemjs\node_modules\es6-module-loader\node_modules\grunt-contrib-uglify\node_modules\maxmin\node_modules\pretty-bytes\node_modules\meow\node_modules\indent-string\node_modules\repeating\node_modules\is-finite\node_modules\*' (OS Error: Системе не удается найти указанный путь.

, errno = 3)
Stack Trace:
dart:async/stream_controller.dart 563                         _StreamController.addError
dart:io/directory_impl.dart 384                               _AsyncDirectoryLister.error
dart:io/directory_impl.dart 344                               _AsyncDirectoryLister.next.<fn>
package:stack_trace/src/stack_zone_specification.dart 134:26  StackZoneSpecification.registerUnaryCallback.<fn>.<fn>
package:stack_trace/src/stack_zone_specification.dart 205:15  StackZoneSpecification._run
package:stack_trace/src/stack_zone_specification.dart 134:14  StackZoneSpecification.registerUnaryCallback.<fn>
dart:async/zone.dart 911                                      _rootRunUnary
dart:async/zone.dart 811                                      _CustomZone.runUnary
dart:async/future_impl.dart 494                               _Future._propagateToListeners.handleValueCallback
dart:async/future_impl.dart 577                               _Future._propagateToListeners
dart:async/future_impl.dart 368                               _Future._completeWithValue
dart:async/future_impl.dart 422                               _Future._asyncComplete.<fn>
package:stack_trace/src/stack_zone_specification.dart 205:15  StackZoneSpecification._run
package:stack_trace/src/stack_zone_specification.dart 124:48  StackZoneSpecification.registerCallback.<fn>
dart:async/zone.dart 904                                      _rootRun
dart:async/zone.dart 803                                      _CustomZone.run
dart:async/zone.dart 709                                      _CustomZone.runGuarded
dart:async/zone.dart 734                                      _CustomZone.bindCallback.<fn>
dart:async/schedule_microtask.dart 43                         _microtaskLoop
dart:async/schedule_microtask.dart 52                         _microtaskLoopEntry
dart:isolate-patch/isolate_patch.dart 96                      _runPendingImmediateCallback
dart:isolate-patch/isolate_patch.dart 149                     _RawReceivePortImpl._handleMessage
===== asynchronous gap ===========================
dart:async/zone.dart 836                                      _CustomZone.registerUnaryCallback
dart:async/future_impl.dart 208                               _Future.then
dart:io/directory_impl.dart 326                               _AsyncDirectoryLister.next
dart:io/directory_impl.dart 292                               _AsyncDirectoryLister.onListen.<fn>
package:stack_trace/src/stack_zone_specification.dart 134:26  StackZoneSpecification.registerUnaryCallback.<fn>.<fn>
package:stack_trace/src/stack_zone_specification.dart 205:15  StackZoneSpecification._run
package:stack_trace/src/stack_zone_specification.dart 134:14  StackZoneSpecification.registerUnaryCallback.<fn>
dart:async/zone.dart 911                                      _rootRunUnary
dart:async/zone.dart 811                                      _CustomZone.runUnary
dart:async/future_impl.dart 494                               _Future._propagateToListeners.handleValueCallback
dart:async/future_impl.dart 577                               _Future._propagateToListeners
dart:async/future_impl.dart 368                               _Future._completeWithValue
dart:async/future_impl.dart 422                               _Future._asyncComplete.<fn>
package:stack_trace/src/stack_zone_specification.dart 205:15  StackZoneSpecification._run
package:stack_trace/src/stack_zone_specification.dart 124:48  StackZoneSpecification.registerCallback.<fn>
dart:async/zone.dart 904                                      _rootRun
dart:async/zone.dart 803                                      _CustomZone.run
dart:async/zone.dart 709                                      _CustomZone.runGuarded
dart:async/zone.dart 734                                      _CustomZone.bindCallback.<fn>
dart:async/schedule_microtask.dart 43                         _microtaskLoop
dart:async/schedule_microtask.dart 52                         _microtaskLoopEntry
dart:isolate-patch/isolate_patch.dart 96                      _runPendingImmediateCallback
dart:isolate-patch/isolate_patch.dart 149                     _RawReceivePortImpl._handleMessage
===== asynchronous gap ===========================
dart:async/zone.dart 836                                      _CustomZone.registerUnaryCallback
dart:async/future_impl.dart 208                               _Future.then
dart:io/directory_impl.dart 289                               _AsyncDirectoryLister.onListen
dart:async/stream_controller.dart 777                         _runGuarded
dart:async/stream_controller.dart 656                         _StreamController._subscribe.<fn>
dart:async/stream_impl.dart 414                               _BufferingStreamSubscription._guardCallback
dart:async/stream_controller.dart 655                         _StreamController._subscribe
dart:async/stream_controller.dart 795                         _ControllerStream._createSubscription
dart:async/stream_impl.dart 474                               _StreamImpl.listen
package:watcher/src/directory_watcher/windows.dart 388:39     _WindowsDirectoryWatcher._listDir
package:watcher/src/directory_watcher/windows.dart 92:5       _WindowsDirectoryWatcher._WindowsDirectoryWatcher
package:watcher/src/directory_watcher/windows.dart 26:36      WindowsDirectoryWatcher.WindowsDirectoryWatcher.<fn>
package:watcher/src/resubscribable.dart 49:25                 ResubscribableWatcher.ResubscribableWatcher.<fn>
dart:async/stream_controller.dart 777                         _runGuarded
dart:async/broadcast_stream_controller.dart 212               _BroadcastStreamController._subscribe
dart:async/stream_controller.dart 795                         _ControllerStream._createSubscription
dart:async/stream_impl.dart 474                               _StreamImpl.listen
bin\test.dart 13:39                                           main.<fn>
package:stack_trace/src/chain.dart 75:24                      Chain.capture.<fn>
dart:async/zone.dart 904                                      _rootRun
dart:async/zone.dart 803                                      _CustomZone.run
dart:async/zone.dart 1262                                     runZoned
package:stack_trace/src/chain.dart 73:12                      Chain.capture
bin\test.dart 12:9                                            main
dart:isolate-patch/isolate_patch.dart 261                     _startIsolate.<fn>
dart:isolate-patch/isolate_patch.dart 148                     _RawReceivePortImpl._handleMessage

[lrhn]

Streams do "top-level" uncaught errors. More precisely, if a stream has an unhandled error, it's passed to the stream's zone's error handler. For the top-level zone, that means raising it as an uncaught error in the next microtask.

[sgjesse]

Right now the only way to deal with long paths on Windows is by using "extended-length path", see https://msdn.microsoft.com/en-us/library/windows/desktop/aa365247(v=vs.85).aspx. Extended-length paths can be up to about 32k characters, and use the special \\?\ prefix. They must use backslash as separator and must be absolute. Relative paths are always limited to 260 characters for Win32 calls.

The following example creates and lists long paths:

import 'dart:io';

Future createDeepDirectory(String path) async {
  for (int i = 0; i < 100; i++) {
    await new Directory(path).create();
    path += '\\0123456789';
  }
}

main() async {
  var d = new Directory('\\\\?\\D:\\src\\dart\\github\\sdk\\sdk\\deep');
  await createDeepDirectory(d.path);
  d.listSync(recursive: true).forEach((e) => print(e.path.length));
}

We have discussed making behind the scenes conversion to extended-length path in dart:io on Windows, but I am not sure this can be done fully transparent.

[scheglov]

@sgjesse OK, thank you for information.

I'm sure it is better to do in one place - in dart:io than to have all the applications potentially buggy on Windows, or to implement conversion in every application separately.

[bwilkerson]

ping. This is still causing problems for users of analysis server, and we really need to get this fixed.

[eseidel]

We stumbled on this bug (for other reasons) today. Our bug ended up not related, but it did make me wonder if this not-commented-on-in-8-years bug was still useful or should be closed?

[mraleph]

We have been slowly fixing issues with long file paths on Windows. So we should be able to fix this one as well. We should just make sure that we force file path to long form eagerly (similar to I changed it to happen for recursive deletion).

cc @aam