DCLI is broken for Dart 3.5

jifferon commented 2 months ago

Hi! Unfortunately, DCLI 6.0.1 (and 6.0.0) is broken on Dart 3.5

When I do dcli compile --install

This is the error log:

ArgumentError: Invalid argument(s): Couldn't resolve native function 'pthread_mutex_timedlock' in 'package:native_synchronization_temp/src/bindings/pthread.dart' : No asset with id 'package:native_synchronization_temp/src/bindings/pthread.dart' found. No available native assets. Attempted to fallback to process lookup. dlsym(RTLD_DEFAULT, pthread_mutex_timedlock): symbol not found.

Stacktrace: dart:ffi                                                            Native._ffi_resolver_function
package:native_synchronization_temp/src/bindings/pthread.dart       pthread_mutex_timedlock
package:native_synchronization_temp/posix.dart 58:20                _PosixMutex._timedLock
package:native_synchronization_temp/posix.dart 44:7                 
_PosixMutex._lock
package:native_synchronization_temp/primitives.dart 75:5            
Mutex.runLocked
package:native_synchronization_temp/mailbox.dart 109:19             Mailbox._takeTimed
package:native_synchronization_temp/mailbox.dart 100:14             
Mailbox.take
package:dcli/src/util/runnable_process.dart 341:38                  RunnableProcess.start
package:dcli/src/util/runnable_process.dart 194:7                   RunnableProcess.run
package:dcli/src/functions/run.dart 180:19                          
startFromArgs

ERROR: package:dcli/src/script/dart_sdk.dart 247:7                         DartSdk.runPub
package:dcli/src/script/dart_sdk.dart 381:5                         
DartSdk.runPubGet
package:dcli/src/script/pub_get.dart 31:17                          
PubGet.run
package:dcli/src/script/dart_project.dart 354:14                    DartProject._pubget.<fn>
package:dcli/src/util/named_lock.dart 125:17                        NamedLock.withLockAsync
package:dcli/src/script/dart_project.dart 346:38                    DartProject._pubget
package:dcli/src/script/dart_project.dart 257:21                    DartProject.warmup.<fn>
package:dcli/src/util/named_lock.dart 125:17                        NamedLock.withLockAsync
package:dcli/src/script/dart_project.dart 235:5                     DartProject.warmup
package:dcli_sdk/src/commands/compile.dart 109:10                   CompileCommand.compileScript
package:dcli_sdk
ERROR: /src/commands/compile.dart 178:20                   CompileCommand.compileScripts
package:dcli_sdk/src/commands/compile.dart 65:7                     CompileCommand.run
package:dcli_sdk/src/script/command_line_runner.dart 101:11         CommandLineRunner.process
package:dcli_sdk/src/script/entry_point.dart 43:18                  EntryPoint._parseCmdLine
../../.pub-cache/hosted/pub.dev/dcli_sdk-6.0.0/bin/dcli.dart 17:22  DCli.run
../../.pub-cache/hosted/pub.dev/dcli_sdk-6.0.0/bin/dcli.dart 12:3   main

dcli doctor is:

DCli version      6.0.1                                                  

os                macos                                                  
os version        Version 14.5 (Build 23F79)
path separator    /                                                      

dart version      3.5.0                                                  

dcli path         /<HOME>/.pub-cache/bin/dcli                            
dart exe path     /<HOME>/flutter/bin/dart                               
dart path         /<HOME>/flutter/bin/dart                                which: /<HOME>/flutter/bin/dart                        
compiler          using 'dart compile exe'                               

pub               using 'dart pub'                                       

pub cache         /<HOME>/.pub-cache                                     
PUB_CACHE Env     false                                                  

package config    /<HOME>/not passed

bsutton commented 2 months ago

I should have a fix available tomorrow

On Tue, 13 Aug 2024, 6:37 pm jifferon, @.***> wrote:

Hi! Unfortunately, DCLI 6.0.1 (and 6.0.0) is broken on Dart 3.5

When I do dcli compile --install

This is the error log:

ArgumentError: Invalid argument(s): Couldn't resolve native function 'pthread_mutex_timedlock' in 'package:native_synchronization_temp/src/bindings/pthread.dart' : No asset with id 'package:native_synchronization_temp/src/bindings/pthread.dart' found. No available native assets. Attempted to fallback to process lookup. dlsym(RTLD_DEFAULT, pthread_mutex_timedlock): symbol not found.

Stacktrace: dart:ffi Native._ffi_resolver_function package:native_synchronization_temp/src/bindings/pthread.dart pthread_mutex_timedlock package:native_synchronization_temp/posix.dart 58:20 _PosixMutex._timedLock package:native_synchronization_temp/posix.dart 44:7 _PosixMutex._lock package:native_synchronization_temp/primitives.dart 75:5 Mutex.runLocked package:native_synchronization_temp/mailbox.dart 109:19 Mailbox._takeTimed package:native_synchronization_temp/mailbox.dart 100:14 Mailbox.take package:dcli/src/util/runnable_process.dart 341:38 RunnableProcess.start package:dcli/src/util/runnable_process.dart 194:7 RunnableProcess.run package:dcli/src/functions/run.dart 180:19 startFromArgs

ERROR: package:dcli/src/script/dart_sdk.dart 247:7 DartSdk.runPub package:dcli/src/script/dart_sdk.dart 381:5 DartSdk.runPubGet package:dcli/src/script/pub_get.dart 31:17 PubGet.run package:dcli/src/script/dart_project.dart 354:14 DartProject._pubget. package:dcli/src/util/named_lock.dart 125:17 NamedLock.withLockAsync package:dcli/src/script/dart_project.dart 346:38 DartProject._pubget package:dcli/src/script/dart_project.dart 257:21 DartProject.warmup. package:dcli/src/util/named_lock.dart 125:17 NamedLock.withLockAsync package:dcli/src/script/dart_project.dart 235:5 DartProject.warmup package:dcli_sdk/src/commands/compile.dart 109:10 CompileCommand.compileScript package:dcli_sdk ERROR: /src/commands/compile.dart 178:20 CompileCommand.compileScripts package:dcli_sdk/src/commands/compile.dart 65:7 CompileCommand.run package:dcli_sdk/src/script/command_line_runner.dart 101:11 CommandLineRunner.process package:dcli_sdk/src/script/entry_point.dart 43:18 EntryPoint._parseCmdLine ../../.pub-cache/hosted/pub.dev/dcli_sdk-6.0.0/bin/dcli.dart 17:22 DCli.run ../../.pub-cache/hosted/pub.dev/dcli_sdk-6.0.0/bin/dcli.dart 12:3 main

dcli doctor is:

DCli version 6.0.1

os macos os version Version 14.5 (Build 23F79) path separator /

dart version 3.5.0

dcli path //.pub-cache/bin/dcli dart exe path //flutter/bin/dart dart path //flutter/bin/dart which: //flutter/bin/dart compiler using 'dart compile exe'

pub using 'dart pub'

pub cache //.pub-cache PUB_CACHE Env false

package config //not passed

— Reply to this email directly, view it on GitHub https://github.com/onepub-dev/dcli/issues/252, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAG32OEKAUNT6VG5H2W3L5LZRHAU3AVCNFSM6AAAAABMNYUM3KVHI2DSMVQWIX3LMV43ASLTON2WKOZSGQ3DENZUHE4DKMA . You are receiving this because you are subscribed to this thread.Message ID: @.***>

dw2kim commented 2 months ago

+1

bsutton commented 2 months ago

I've published dcli_sdk 6.0.2 which should resolve this issue.

danieletulone commented 2 months ago

Hi, I have same problem. dcli version is 6.0.5 and dart 3.4.3, on macos arm64

ArgumentError: Invalid argument(s): Couldn't resolve native function 'pthread_mutex_timedlock' in 'package:native_synchronization_temp/src/bindings/pthread.dart' : No asset with id 'package:native_synchronization_temp/src/bindings/pthread.dart' found. No available native assets. Attempted to fallback to process lookup. dlsym(RTLD_DEFAULT, pthread_mutex_timedlock): symbol not found.

bsutton commented 2 months ago

Can you updated to 3.5?

On Sun, 18 Aug 2024, 11:49 pm Daniele Tulone, @.***> wrote:

Hi, I have same problem. dcli version is 6.0.5 and dart 3.4.3, on macos arm64

— Reply to this email directly, view it on GitHub https://github.com/onepub-dev/dcli/issues/252#issuecomment-2295270705, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAG32OFJ2PRP4WURVH32I3LZSCQ77AVCNFSM6AAAAABMNYUM3KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEOJVGI3TANZQGU . You are receiving this because you modified the open/close state.Message ID: @.***>

danieletulone commented 2 months ago

@bsutton Yes, I updated to 3.5 (3.5.1), the error still remains.

bsutton commented 2 months ago

So in meant to guess what the error is?

On Mon, 19 Aug 2024, 9:12 am Daniele Tulone, @.***> wrote:

@bsutton https://github.com/bsutton Yes, I updated to 3.5 (3.5.1), the error still remains.

— Reply to this email directly, view it on GitHub https://github.com/onepub-dev/dcli/issues/252#issuecomment-2295428795, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAG32OAPNK34HUVWRUTWEGDZSES4HAVCNFSM6AAAAABMNYUM3KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEOJVGQZDQNZZGU . You are receiving this because you were mentioned.Message ID: @.***>

danieletulone commented 2 months ago

@bsutton The solution seems to be to provide a custom implementation of the pthread_mutex_timedlock function. I'm making some tries editing directly the pub content.

I don't known low level details so with the help of chatgpt I replaced pthread_mutex_timedlock function in native_synchronization_temp/lib/src/bindings/pthread.dart with:

/// at the top, add another import
import 'package:ffi/ffi.dart';

/// Constants (maybe we should get these values in another way, currently hard coded)
const int EBUSY = 16; // Resource busy
const int ETIMEDOUT = 60; // Operation timed out
const int CLOCK_REALTIME = 0;

/// Bind the `clock_gettime` function from the C library
final clock_gettime = DynamicLibrary.process().lookupFunction<
    Int32 Function(Int32 clockId, Pointer<pthread_timespec_t> tp),
    int Function(int clockId, Pointer<pthread_timespec_t> tp)>('clock_gettime');

/// Mutex lock implementation for macOS
int pthread_mutex_timedlock(
    Pointer<pthread_mutex_t> mutex, Pointer<pthread_timespec_t> abstime) {
  final pthreadMutexTryLock = DynamicLibrary.process().lookupFunction<
      Int Function(Pointer<pthread_mutex_t>),
      int Function(Pointer<pthread_mutex_t>)>('pthread_mutex_trylock');

  final nanosleep = DynamicLibrary.process().lookupFunction<
      Int Function(
          Pointer<pthread_timespec_t> req, Pointer<pthread_timespec_t> rem),
      int Function(Pointer<pthread_timespec_t> req,
          Pointer<pthread_timespec_t> rem)>('nanosleep');

  final timeout = abstime.ref;
  final now = malloc<pthread_timespec_t>();
  final remainingTime = malloc<pthread_timespec_t>();

  while (true) {
    final ret = pthreadMutexTryLock(mutex);
    if (ret == 0) {
      malloc.free(now);
      malloc.free(remainingTime);
      return 0;
    } else if (ret != EBUSY) {
      malloc.free(now);
      malloc.free(remainingTime);
      return ret;
    }

    clock_gettime(CLOCK_REALTIME, now);

    if (now.ref.tv_sec > timeout.tv_sec ||
        (now.ref.tv_sec == timeout.tv_sec &&
            now.ref.tv_nsec >= timeout.tv_nsec)) {
      malloc.free(now);
      malloc.free(remainingTime);
      return ETIMEDOUT;
    }

    remainingTime.ref.tv_sec = timeout.tv_sec - now.ref.tv_sec;
    remainingTime.ref.tv_nsec = timeout.tv_nsec - now.ref.tv_nsec;
    if (remainingTime.ref.tv_nsec < 0) {
      remainingTime.ref.tv_sec -= 1;
      remainingTime.ref.tv_nsec += 1000000000;
    }

    nanosleep(remainingTime, nullptr);
  }
}

Now the cli works, I have tried with a 'some command'.run (that previously didn't work)

I don't know if the provided code is a right mock implementation of the pthread_mutex_timedlock.

bsutton commented 2 months ago

Check what version of dcli was imported. I suspect you had an old version in your machine.

Delete your pub cache and see if this fixes the issue.

On Mon, 19 Aug 2024, 10:26 am Daniele Tulone, @.***> wrote:

@bsutton https://github.com/bsutton The solution seems to be to provide a custom implementation of the pthread_mutex_timedlock function. I'm making some tries editing directly the pub content.

I don't known low level details so with the help of chatgpt I have replaced the function of native_synchronization_temp/lib/src/bindings/pthread.dart with:

/// at the top, add another importimport 'package:ffi/ffi.dart'; /// Constants (maybe we should get these values in another way, currently hard coded)const int EBUSY = 16; // Resource busyconst int ETIMEDOUT = 60; // Operation timed outconst int CLOCK_REALTIME = 0; /// Bind the clock_gettime function from the C libraryfinal clock_gettime = DynamicLibrary.process().lookupFunction< Int32 Function(Int32 clockId, Pointer tp), int Function(int clockId, Pointer tp)>('clock_gettime'); /// Mutex lock implementation for macOSint pthread_mutex_timedlock( Pointer mutex, Pointer abstime) { final pthreadMutexTryLock = DynamicLibrary.process().lookupFunction< Int Function(Pointer), int Function(Pointer)>('pthread_mutex_trylock');

final nanosleep = DynamicLibrary.process().lookupFunction< Int Function( Pointer req, Pointer rem), int Function(Pointer req, Pointer rem)>('nanosleep');

final timeout = abstime.ref; final now = malloc(); final remainingTime = malloc();

while (true) { final ret = pthreadMutexTryLock(mutex); if (ret == 0) { malloc.free(now); malloc.free(remainingTime); return 0; } else if (ret != EBUSY) { malloc.free(now); malloc.free(remainingTime); return ret; }
clock_gettime(CLOCK_REALTIME, now);

if (now.ref.tv_sec > timeout.tv_sec ||
    (now.ref.tv_sec == timeout.tv_sec &&
        now.ref.tv_nsec >= timeout.tv_nsec)) {
  malloc.free(now);
  malloc.free(remainingTime);
  return ETIMEDOUT;
}

remainingTime.ref.tv_sec = timeout.tv_sec - now.ref.tv_sec;
remainingTime.ref.tv_nsec = timeout.tv_nsec - now.ref.tv_nsec;
if (remainingTime.ref.tv_nsec < 0) {
  remainingTime.ref.tv_sec -= 1;
  remainingTime.ref.tv_nsec += 1000000000;
}

nanosleep(remainingTime, nullptr);
} }

Now the cli works, I have tried with a 'some command'.run (that previously didn't work)

I don't know if the provided code is a right mock implementation of the pthread_mutex_timedlock.

— Reply to this email directly, view it on GitHub https://github.com/onepub-dev/dcli/issues/252#issuecomment-2295458470, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAG32ODSAV4DHI34ITANZ5LZSE3U5AVCNFSM6AAAAABMNYUM3KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEOJVGQ2TQNBXGA . You are receiving this because you were mentioned.Message ID: @.***>

bsutton commented 2 months ago

So I believe that what happened is:

when I released dcli 6.0.5 I didn't update the minimum constraint on the dcli library for dcli_sdk.

On a clean system dcli_sdk links to dcli 6.0.5 but I think on a system with an earlier version of dcli, that dart links the dcli_sdk to the old resident version of dcli.

So I've now update the min version for dcli in dcli_sdk which I hope resolves this issue.

I've released dcli_sdk 6.0.5 to pub.dev with the fix.

let me know how this goes.

bsutton commented 2 months ago

And now I've just read your updated comment and realised what the code was for.

So you are saying that macos is missing the pthread_mutex_timedlock. That puts quite a different shade on the problem.

So I don't have time now but I will look at the practicalities of integrating your mutex code into the library.

At a quick look I don't think the chat gpt implementation is correct.

This link proposes a solution:

https://lists.apache.org/thread/yqdnsky6svc5cfdkx1s8m4d3j8h9xyj6

edit: updated link

danieletulone commented 2 months ago

So, the issue is more related to native_synchronization_temp/native_synchronization?

bsutton commented 2 months ago

@danieletulone I've move this discussion to a new issue as its a separate problem.

https://github.com/onepub-dev/dcli/issues/253

onepub-dev / dcli

DCLI is broken for Dart 3.5 #252