dart pub get fails with: pubspec.lock file has changed since the xxx/.dart_tool/package_config.json file was generated, please run "dart pub get" again

[bsutton]

Environment

dart --version
Dart SDK version: 2.18.0 (stable) (Fri Aug 26 10:22:54 2022 +0000) on "linux_x64"

I'm running a zfs file system on a fairly beefy PC.

Processor: AMD® Ryzen 7 3700x 8-core processor × 16

Memory: 32GB

The file system is a pair of SSD.

Problem

Since I've upgraded to dart 2.18 I've been getting re-occurring problems with the error:

pubspec.lock file has changed since the xxx/.dart_tool/package_config.json file was generated, please run "dart pub get" again

I believe that I've tracked down this problem to a time precision issue.

After running dart pub get I see the following times:

pubspec.lock:                 2022-09-07 16:59:34.000527000 +1000
package_config.json:     2022-09-07 16:59:34.000532000 +1000

Note the tiny separation between these two timestamps.

If I run the following dart code which obtains the timestamp of each and prints it:

void main(List<String> args) {
  final path = join(HOME, 'git', 'dcli', 'dcli');

  final pubspecModified = File(join(path, 'pubspec.lock')).lastModifiedSync();
  final lockFileModified =
      File(join(path, '.dart_tool', 'package_config.json')).lastModifiedSync();

  print('lock: $pubspecModified');
  print('config: $lockFileModified');
}

I get:

lock: 2022-09-07 16:59:34.000
config: 2022-09-07 16:59:34.000

As you can see the call to lastmodified has lost the precisions.

From the pub command the problematic code is: lib/src/entrypoint.dart

 var pubspecModified = File(pubspecPath).lastModifiedSync();
    var lockFileModified = File(lockFilePath).lastModifiedSync();

    var pubspecChanged = lockFileModified.isBefore(pubspecModified);

The call to isBefore is going to fail as the two timestamps are equal.

This is a show stopper for me as I'm unable to run my unit tests as the consistently fail with this error.

[sigurdm]

Maybe it is enough to do:

var pubspecChanged = lockFileModified.isBefore(pubspecModified) || lockFileModified ==  pubspecModified;

[bsutton]

Sounds like it would solve the problem.

On Thu, 22 Sept 2022, 6:42 pm Sigurd Meldgaard, @.***> wrote:

Maybe it is enough to do:

var pubspecChanged = lockFileModified.isBefore(pubspecModified) || lockFileModified == pubspecModified;

— Reply to this email directly, view it on GitHub https://github.com/dart-lang/pub/issues/3558#issuecomment-1254712125, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAG32OCXFCEF4U3VZC67LE3V7QLWRANCNFSM6AAAAAAQGPAB2Y . You are receiving this because you authored the thread.Message ID: @.***>

[sigurdm]

Looking at this again.

The mentioned error message comes from https://github.com/dart-lang/pub/blob/6e769c086115dfc46d6cb593390d79908aec749f/lib/src/entrypoint.dart#L795

Here we are guarded, not by time stamps, but by an actual check of the contents of .dart_tool/package_config.json

I wonder if something else is going on...

[bsutton]

Any update on this. I'm still seeing this problem on dart 2.19.1

[bsutton]

You should be able to reproduce the problem by setting both files to the exact same 'last modified' time.

[sxd1140]

I have precisions issue too. lastModified precisions only to seconds.

[sigurdm]

lastModified precisions only to seconds.

I think this is the culprit! The setLastModified functionality in dart is appearently broken. Started a fix: https://dart-review.googlesource.com/c/sdk/+/289503

[bsutton]

The nano-seconds was computed wrong on non-windows platforms Windows only supported down to 1-second granularity.

Does this suggest we still have a problem, at least on windows? If the granularity on any platform is 1 second then we will still need your suggested alteration of:

var pubspecChanged = lockFileModified.isBefore(pubspecModified) || lockFileModified == pubspecModified;

[sxd1140]

Windows only supported down to 1-second granularity.

https://learn.microsoft.com/en-us/windows/win32/sysinfo/file-times Seem like Windows provide 100-nanosecond

[sigurdm]

The problem (if I am correct in my belief) is a bug in the implementation of File.setLastModified that it would set the nanoseconds to one 1000th of what it really is. (while the seconds were set correctly).

On windows the granularity of File.setLastModified was just 1 second.

[bsutton]

I don't think that fits the data I original posted

lock: 2022-09-07 16:59:34.000

The data suggest a complete zeroing of nanoseconds.

[sigurdm]

And that is on linux?

In my experiments with the fix, it seems some filesystems don't preserve the full nanosecond precision, so the nanoseconds might be cut off by the filesystem?

[bsutton]

Linux with a zfs filesystem.

On Fri, 24 Mar 2023, 7:52 pm Sigurd Meldgaard, @.***> wrote:

And that is on linux?

In my experiments with the fix, it seems some filesystems don't preserve the full nanosecond precision, so the nanoseconds might be cut off by the filesystem?

— Reply to this email directly, view it on GitHub https://github.com/dart-lang/pub/issues/3558#issuecomment-1482453181, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAG32OFHFFIBNU2CX43BYE3W5VOFNANCNFSM6AAAAAAQGPAB2Y . You are receiving this because you authored the thread.Message ID: @.***>

[bsutton]

If its just a reduction in precision then i would still expect the milliseconds to be non zero.

On Fri, 24 Mar 2023, 7:53 pm Brett Sutton, @.***> wrote:

Linux with a zfs filesystem.

On Fri, 24 Mar 2023, 7:52 pm Sigurd Meldgaard, @.***> wrote:

And that is on linux?

In my experiments with the fix, it seems some filesystems don't preserve the full nanosecond precision, so the nanoseconds might be cut off by the filesystem?

— Reply to this email directly, view it on GitHub https://github.com/dart-lang/pub/issues/3558#issuecomment-1482453181, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAG32OFHFFIBNU2CX43BYE3W5VOFNANCNFSM6AAAAAAQGPAB2Y . You are receiving this because you authored the thread.Message ID: @.***>

[bsutton]

test from my linux zfs system:

void main() async {
  final file = File('fred.txt');
  await file.create();

  print(file.statSync().modified);

  sleep(const Duration(seconds: 5));

  await file.setLastModified(DateTime.now());

  print(file.statSync().modified);

}

output

2023-03-24 19:57:18.668
2023-03-24 20:01:02.000

So setLastModified appears to truncate the nanoseconds completely.

[sigurdm]

Yeah, I agree.

Could you try to run this program:

import 'dart:io';

main() async {
  final f = File('test-file');
  f.writeAsStringSync('hi');
  f.setLastModifiedSync(DateTime(2018, 1, 2, 3, 4, 5, 6, 7));
}

And then stat test-file.

I get:

Modify: 2018-01-02 03:04:05.000006000 +0000

[sigurdm]

Try to use stat from the command line. Dart DateTime only has microsecond precision.

[bsutton]

I think we crossed the streams:

stat test-file
  File: test-file
  Size: 2               Blocks: 2          IO Block: 512    regular file
Device: 2bh/43d Inode: 943642      Links: 1
Access: (0644/-rw-r--r--)  Uid: ( 1000/ bsutton)   Gid: (  888/  shared)
Access: 2023-03-24 20:03:01.940279550 +1100
Modify: 2018-01-02 03:04:05.000006000 +1100
Change: 2023-03-24 20:03:01.952280158 +1100
 Birth: 2023-03-24 20:03:01.940279550 +1100

[bsutton]

so modified is completely borked.

edit: and now I realized you set the date to 2018 :D

[sigurdm]

Only when we do the touch from inside dart. Still I'm not sure about the root issue here is caused by that.

dart pub get should not even check the timestamps and ask you to run dart pub get again. We do that in assertUpToDate that should not be called from pub get. It calls instead acquireDependencies.

Is it dart pub deps that is failing?