Line 935 Error - KeyError: 'global_x_train_shape'

[Likelyt]

https://github.com/Declancharrison/Level-Set-Boosting/blob/30643e959a9fd164d278cd16d91dfc2c660e6d44/LSBoost.py#L935

I tried to locally run it over Mac of ipynb file. However, it shows that _"KeyError: 'global_x_trainshape'" in line 935 of LSBoost.py file. Do you know how to solve it?

Thank!

[Declancharrison]

Hm. That probably means the shared memory array failed to initialize. Can you post the whole error code? Also, did this occur while fitting the model or predicting with a fit model?

[Likelyt]

When I run the "LSBoostReg.fit(x_train[:10000], y_train[:10000])". It shows this:

LS: 0%| 0/100 ?, Training Error=0.0496468 Memory released!

RemoteTraceback Traceback (most recent call last) RemoteTraceback: """ Traceback (most recent call last): File "/Users/liyt/opt/anaconda3/envs/PE/lib/python3.8/multiprocessing/pool.py", line 125, in worker result = (True, func(*args, **kwds)) File "/Users/liyt/opt/anaconda3/envs/PE/lib/python3.8/multiprocessing/pool.py", line 51, in starmapstar return list(itertools.starmap(args[0], args[1])) File "/Users/liyt/Documents/2023Spring/BIC/multi-calibration/code/Level-Set-Boosting-main/LSBoost.py", line 935, in worker x_train_shared = np.ndarray(globals()['global_x_train_shape'], dtype = np.float64, buffer=globals()['global_x_train_shm'].buf) KeyError: 'global_x_train_shape' """

The above exception was the direct cause of the following exception:

KeyError Traceback (most recent call last) /var/folders/kx/qstn8wln7mv3jp6yrn13wff00000gn/T/ipykernel_72960/411476469.py in 2 # y_train = y_train[:1000], (437816,) 3 # use a subset of the data to speed up the process ----> 4 LSBoostReg.fit(x_train[:10000], y_train[:10000])

~/Documents/2023Spring/BIC/multi-calibration/code/Level-Set-Boosting-main/LSBoost.py in fit(self, x_train, y_train) 702 # rounds 1 and 2 updates 703 elif ((i == 0) or (i == 1)): --> 704 self.update() 705 continue 706

~/Documents/2023Spring/BIC/multi-calibration/code/Level-Set-Boosting-main/LSBoost.py in update(self) 228 229 # fit weak learners in parallel over non-empty level sets --> 230 result = executor.starmap(worker, zip(occupied_level_sets, repeat(self.round_count))) 231 232 # wait for all computations to terminate and close executor

~/opt/anaconda3/envs/PE/lib/python3.8/multiprocessing/pool.py in starmap(self, func, iterable, chunksize) 370 func and (a, b) becomes func(a, b). 371 ''' --> 372 return self._map_async(func, iterable, starmapstar, chunksize).get() 373 374 def starmap_async(self, func, iterable, chunksize=None, callback=None,

~/opt/anaconda3/envs/PE/lib/python3.8/multiprocessing/pool.py in get(self, timeout) 769 return self._value 770 else: --> 771 raise self._value 772 773 def _set(self, i, obj):

KeyError: 'global_x_train_shape'

[Declancharrison]

It doesn't seem to be mapping the data into a shared memory region. Can you try running the following in a cell:

def create_shared_memory_nparray(data, name, create = False): ''' Creates array in shared memory for parallel processing. ''' d_size = np.dtype(np.float64).itemsize * np.prod(data.shape) try: shm = shared_memory.SharedMemory(create=create, size=d_size, name=f'{name}_buf') except: shm = shared_memory.SharedMemory(create=False, size=d_size, name=f'{name}_buf') return shm

dst = np.ndarray(shape=data.shape, dtype=np.float64, buffer=shm.buf) dst[:] = data[:] globals()[f'global_{name}shm'] = shm globals()[f'shms'].append(name) globals()[f'global{name}_shape'] = data.shape return shm

initialize shared data list

globals()[f'shms'] = []

share training data

create_shared_memory_nparray(x_train,'x_train', True) create_shared_memory_nparray(y_train,'y_train', True)

try to access shared data

x_train_shared = np.ndarray(globals()['global_x_train_shape'], dtype = np.float64, buffer=globals()['global_x_train_shm'].buf)

[Likelyt]

It seems to give me the same error.

KeyError Traceback (most recent call last) /var/folders/kx/qstn8wln7mv3jp6yrn13wff00000gn/T/ipykernel_2668/3331667428.py in 23 create_shared_memory_nparray(y_train,'y_train', True) 24 ---> 25 x_train_shared = np.ndarray(globals()['global_x_train_shape'], dtype = np.float64, buffer=globals()['global_x_train_shm'].buf)

__KeyError: 'global_x_train_shape'__

from multiprocessing import shared_memory
def create_shared_memory_nparray(data, name, create = False):
    '''
    Creates array in shared memory for parallel processing.
    '''
    d_size = np.dtype(np.float64).itemsize * np.prod(data.shape)
    try:
        shm = shared_memory.SharedMemory(create=create, size=d_size, name=f'{name}_buf')
    except:
        shm = shared_memory.SharedMemory(create=False, size=d_size, name=f'{name}_buf')
        return shm

    dst = np.ndarray(shape=data.shape, dtype=np.float64, buffer=shm.buf)
    dst[:] = data[:]
    globals()[f'global_{name}shm'] = shm
    globals()[f'shms'].append(name)
    globals()[f'global{name}_shape'] = data.shape
    return shm

globals()[f'shms'] = []

create_shared_memory_nparray(x_train,'x_train', True)
create_shared_memory_nparray(y_train,'y_train', True)

x_train_shared = np.ndarray(globals()['global_x_train_shape'], dtype = np.float64, buffer=globals()['global_x_train_shm'].buf)

[Declancharrison]

You can try inserting the following into the top of the LSBoost.py file after importing multiprocessing: multiprocessing.set_start_method("fork"). This is a MacOS specific issue, and using 'fork' may cause the program to crash depending on if threading is used. It's worth a shot, and if not I may create a non-parallel version for MacOS or rework the parallel use cases.

[Likelyt]

Hi, I tried the code in the linux system and it works. It seems iOS system is not compatible with the code. Additionally, the graph function has error. Not sure whether the package is obsolete or not. However, it does not affect the whole project. Thanks!