UltraDict dependency 'atomics' is not compatible with MacBook silicon (m1)

[JOEAV]

Version : branch master OS: macOS big sur version 11.6 The scenario: I'm using this module in an algotrading bot app. One mechanism I'm driving with this is helping the bot get quick updates from other process which is responsible of transmitting price updates. The dictionary is a smart move as it is the right tool for the job. Process A fill a dictionary with prices . Process B consume those prices and makes math calculations based on them.

My Issue . As the bot run inside a while loop , it never really exits gracefully but through an interrupt (SIGINT , then SIGTERM) if the Producer of dict(Process A) exit by SIGINT its fine. but if Process B (consumer) exit by SIGINT the dictionary seems to enter a state which you can't clear it even with unlink() and close(). only restart helps with this scenario (checked /dev/shm but /shm does not exist on my hd)

That lad me to try the shared lock mechanism (because I thought it might help with accessing this map with a lock) When I run the code again I was given an error stating "atomics" is not found. after a short pip install atomics I found out they don't have a wheel for Mac arm wheel but only universal one. when running again I get the error of "mech-o:wrong architecture" even if I exclude t "shared lock=true" it keeps throw errors on the same thing. a restart to the computer is the only thing which clears that thing.

I suggest sort this quick as MacBook m1 computers are not that rare and it's actually a quite great library which I'm currently cannot really use :\

[ronny-rentner]

Hi @JOEAV,

thanks for your issue report. It's a pitty that the atomics library is not available on Mac Arm. Maybe you could try to compile it yourselves? Here is the git repo: https://github.com/doodspav/atomics

How are you launching those processes A and B? If you can use the fork method, you would not need the shared_lock=True. Then UltraDict uses a normal multiprocessing.RLock

With which parameters are you running UltraDict exactly?

There are a few cleanup fixes in the dev branch. It might be worth to try them to see if your problem goes away.

In this other issue, I've also explained some ways how to "manually" unlink old UltraDict memory. Please check it out and let me know if that helps: https://github.com/ronny-rentner/UltraDict/issues/16#issuecomment-1200358777

If all of that does not help, what is the exact error message your are getting?

[ronny-rentner]

@JOEAV Btw, according to https://www.python.org/downloads/release/python-391/ it should be possible to run 'universal2' wheels on Apple Silicon for the atomics library.

[JOEAV]

A. this is my constructor : priceUpdates = ultradict({},name='test',buffer_size=1_000_000,auto_unlink=False,recurse=False) (trying to change auto_unlink / recurse didn't really matter B. i'm creating the processes through the main process by the help of executioner

 with concurrent.futures.ProcessPoolExecutor(NUM_CORES) as executor:
        try:
            for i in range(NUM_CORES - 1):
                new_future = executor.submit(stock_navigator_wrapper, 
                STOCK_NAMES_COPY[:PAGES_PER_CORE],priceUpdates)
                futures.append(new_future)

            futures.append(
                executor.submit(
                    stock_navigator_wrapper,
                    STOCK_NAMES_COPY[:PAGES_FOR_FINAL_CORE],
                    priceUpdates
                )
            )

priceUpdate UltraDict is created on main process and then it's memory is being sent to other processes by their ignite function C. it's not spitting errors while installing the atomic universal wheel but yet says "Mech-O.Wrong architecture ", so it seems this wheel isn't helping around. @ronny-rentner

[ronny-rentner]

but if Process B (consumer) exit by SIGINT the dictionary seems to enter a state which you can't clear it even with unlink() and close().

@JOEAV Hmm, are you sure this observation is correct? If the consumer dies and its never writing to the dict, why would it have any effect on the dict state? What happens if you try to call unlink() or close() in your Process A after your Process B has died?

Have you tried the dev branch of UltraDict to see if it helps? There are some improvements if you pass around an UltraDict between processes. I don't know anything about ProcessPoolExecutor but maybe it would help to 'reconnect' to the UltraDict in your child processes, so instead of passing around priceUpdate, in each process you do priceUpdate=ultradict(name='myname'). You don't need to pass around the dict. The connection between them happens through the name.

I also recommend not to set auto_unlink=False. This is telling UltraDict not to clean up the shared memory, so it will be kept until a restart. If you have reached this state that would require a restart, you could also try to manually connect to the UltraDict in a Python REPL like ultra=UltraDict(name='my_last_name') and then run ultra.print_status(). What's the output of this?