Optimization scheme 'n2048_t1.wis' was not found

[moonj94]

I installed fCWT via pypi -> pip install fcwt and I get the following warning when 'fast' is enabled (true). I am on Ubuntu 22.04.

WARNING: Optimization scheme 'n2048_t1.wis' was not found, fallback to calculation without optimization.

I am dealing with some data that could benefit a great deal from optimization and speed. How can I fix this issue?

Thanks very much in advance.

[felixdollack]

@moonj94 check the tutorial.ipynb from the cell that says "Optimization plans" This will generate the file that you are missing (you need to set the threads to 1 to get a file with *_t1.wis)

Hope that helps :)

[moonj94]

Hi @felixdollack for getting back! I must have missed that part of the tutorial, thanks for pointing me to it. I generated the necessary files and created my own class to pass through an optimized fcwt object. However, for some reason it is causing my kernel to crash instantly. My nthreads is 1 at the moment.

Here is the crash error message:

Canceled future for execute_request message before replies were done

The Kernel crashed while executing code in the the current cell or a previous cell. Please review the code in the cell(s) to identify a possible cause of the failure. Click [here](https://aka.ms/vscodeJupyterKernelCrash) for more info. View Jupyter [log](command:jupyter.viewOutput) for further details.

I usually get this message when my computer runs out of memory, but I have 64GB RAM. Is there anything else I can do? Any help would be appreciated!

Thanks!

[felixdollack]

@moonj94 the error message sounds like you are using the processing in an async way?

Canceled future for execute_request message before replies were done

If not, then maybe there is some issue with the linux implementation. In that case @fastlib should have a look at this.

[moonj94]

@felixdollack Hmm, I just tried checking if the error persists after turning fast=False, and it does. The problem seems to be when I call self.fcwt_optim.cwt(...). Weird thing is that it should be fairly identical to the boilerplate version of fcwt.cwt. I think the issue was how I made the class. When I put the initialization and actual usage of the optimzation object into one function the kernel doesn't crash, but I now get this error: TypeError: in method 'FCWT_create_FFT_optimization_plan', argument 2 of type 'int'

EDIT I increased the signal length and the optimization plans are being initailized but for some reason not recognized/found. I suppose this leads me to an important question: How does enabling optimization work for fCWT? Is the only way to use optmization to utilize the initailized fcwt_optim object? Or does simply running create_FFT_optimization_plan create the e.g. n4095t1 files? And if so how can I help the code find these objects?

[felixdollack]

The error you mentioned sounds like the max_size argument was not an integer. From what I understand the max_size argument needs to be larger than 1024 (if it is less or equal to 1024 you should see an error different from the one you saw).

To enable optimization you need to set the use_optimization_plan to True when instantiating the FCWT object. To create the optimization files you need to run the method create_FFT_optimization_plan with the FCWT object that has optimization enabled. If you execute it on another instance without optimization enabled it will do nothing (no files are created).

Each FCWT instance is supposed to look for the created wisdom files (if optimization was enabled for this specific instance).

Could you post the class you wrote to get another pair of eyes checking what could be the culprit?

[moonj94]

Here is the class:

class MyCWT(object):
    def __init__(self,scaling=config.scaling,signal_length=100000):
        super().__init__()
        morl = fcwt.Morlet(2.0)
        # nthreads = 1
        # use_normalization=True
        #create a new fcwt object
        nthreads=config.nthreads
        fs,f0,f1,fn=config.fs,config.f0,config.f1,config.fn
        # scaling='log'
        use_optimization_plan=True
        use_normalization=True
        if scaling == "lin":
            self.scales = Scales(morl,FCWT_LINFREQS,fs,f0,f1,fn)
        elif scaling == "log":
            self.scales = Scales(morl,FCWT_LOGSCALES,fs,f0,f1,fn)
        else:
            self.scales = Scales(morl,FCWT_LOGSCALES,fs,f0,f1,fn)

        self.fcwt_optim = fcwt.FCWT(morl, nthreads, use_optimization_plan, use_normalization)
        # signal_length=100000
        if config.fast:
            self.fcwt_optim.create_FFT_optimization_plan(signal_length,'FFTW_ESTIMATE')
        self.fn=fn
    def cwt(self,input):
        if input.ndim > 1:
            raise ValueError("Input must be a vector")

        #check if input is single precision and change to single precision if not
        if input.dtype != 'single':
            input = input.astype('single')
        output = np.zeros((self.fn,input.size), dtype='csingle')
        freqs = np.zeros((self.fn), dtype='single')

        # input=map(float,input)
        # input=input.astype(float)
        self.fcwt_optim.cwt(input,self.scales,output)
        self.scales.getFrequencies(freqs)

        return freqs, output

This class actually causes the kernel to crash for some reason.

So instead I tried to make a function that instantiates a new object everytime it's called. The code didn't crash, but it didn't end up using the optimization for some reason:

def cwtfunc(input, fs=config.fs, f0=config.f0, f1=config.f1, fn=config.fn, nthreads=1, scaling=config.scaling, fast=config.fast, norm=True):
    input_sz=list(np.shape(input))
    signal_length=np.max(np.array(input_sz))
    print(signal_length)
    #check if input is array and not matrix
    if input.ndim > 1:
        raise ValueError("Input must be a vector")

    #check if input is single precision and change to single precision if not
    if input.dtype != 'single':
        input = input.astype('single')

    morl = Morlet(2.0) #use Morlet wavelet with a wavelet-parameter of 2.0

    #Generate scales

    if scaling == "lin":
        scales = Scales(morl,FCWT_LINFREQS,fs,f0,f1,fn)
    elif scaling == "log":
        scales = Scales(morl,FCWT_LOGSCALES,fs,f0,f1,fn)
    else:
        scales = Scales(morl,FCWT_LOGSCALES,fs,f0,f1,fn)

    fcwt = FCWT(morl, nthreads, fast, norm)
    if config.fast:
        fcwt.create_FFT_optimization_plan(signal_length,"FFTW_MEASURE")
    output = np.zeros((fn,input.size), dtype='csingle')
    freqs = np.zeros((fn), dtype='single')

    fcwt.cwt(input,scales,output)
    scales.getFrequencies(freqs)

    return freqs, output

[moonj94]

@felixdollack Wanted to follow up on this. Thanks for your recommendations but I believe I am alraedy doing that. I instantiate a optimization=true fcwt object and then I create the fft optimization plan using that object. But I still get the warning that the wisdom file cannot be found. I am not currently using the class above at the moment as I get a separate error.

[felixdollack]

@moonj94 I have currently no linux to test this so I am doing it on macOS.

... When I put the initialization and actual usage of the optimzation object into one function the kernel doesn't crash, but I now get this error: TypeError: in method 'FCWT_create_FFT_optimization_plan', argument 2 of type 'int'

The reason for this is that you are passing a numpy.int64 instead of int.

In your function cwtfunc you need to wrap you signal_length calculation with int(...)

signal_length=int(np.max(np.array(input_sz)))

Alternatively length(input_sz) should do the same as the first two lines in your function.

Using the function works for me and it also finds and uses the optimization files when config.fast=True.

However, using the class produces a segmentation fault. I don't know exactly why yet, but I will check now

EDIT: The crash seems related to the Scales. If I move it into the cwt function of the class it works.

[felixdollack]

@moonj94 Both the problem and the solution are simple. Since you are working with C-objects that you can only access as pointers, you need to keep a reference to all objects that are needed (also by other objects). Here the Scales need a reference to the Morlet object. Since you only create it during init and it is already destroyed when you are passing Scales to the cwt function it crashes.

Change your code so it uses self.morlet and it will run without crash!

[bjunor]

In my case, I believe I have followed the correct prescription to generate the .wis file namely: ... Threads:8 Calculating optimal scheme for forward FFT with N:65536 Calculating optimal scheme for backward FFT with N:65536 Optimization schemes for N: 65536 have been calculated. Next time you use fCWT it will automatically choose the right optimization scheme based on number of threads and signal length. Threads:8 Calculating optimal scheme for forward FFT with N:131072 Calculating optimal scheme for backward FFT with N:131072 Optimization schemes for N: 131072 have been calculated. Next time you use fCWT it will automatically choose the right optimization scheme based on number of threads and signal length. WARNING: Optimization scheme 'n2097152_t8.wis' was not found, fallback to calculation without optimization.

but, as you can see, the .wis file isn't found. What am I missing?

[felixdollack]

Hey @bjunor, it seems that your optimization is not covering your use case. Your log only shows optimization until N: 131072, but your input seems to be in the order of N: 2097152.

Can you increase the signal length used in the optimization?

# signal_length should be as long as your actual signal to get the correct optimization files
create_FFT_optimization_plan(signal_length,'FFTW_ESTIMATE')

[bjunor]

Ah. OK. I will try this. I did experiment with the size of the window --- but in the opposite direction i.e. smaller, because I thought that the code might have hit some (possibly arbitrary) limit. I thought that constraining/defining the scales would take care of the optimization window.

Bill

---

From: Felix @.> Sent: Saturday, May 20, 2023 1:24 AM To: fastlib/fCWT @.> Cc: bjunor @.>; Mention @.> Subject: Re: [fastlib/fCWT] Optimization scheme 'n2048_t1.wis' was not found (Issue #30)

Hey @bjunorhttps://github.com/bjunor, it seems that your optimization is not covering your use case. Your log only shows optimization until N: 131072, but your input seems to be in the order of N: 2097152.

Can you increase the signal length used in the optimization?

signal_length should be as long as your actual signal to get the correct optimization files

create_FFT_optimization_plan(signal_length,'FFTW_ESTIMATE')

— Reply to this email directly, view it on GitHubhttps://github.com/fastlib/fCWT/issues/30#issuecomment-1555782586, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ALNITGJ44CATXPYXQUC7MU3XHBWSVANCNFSM6AAAAAAXDMPBRI. You are receiving this because you were mentioned.Message ID: @.***>

[bjunor]

Thanks to Felix for the solution. I had not specified the window for optimization correctly. My record length is 1,440,000 but I had left the window at the coded default of 100,000. I did experiment with this previously but I had thought that the window length needed would be defined by the range of scales requested --- and so I had constrained the scales but still couldn't get the optimization wisdom file to be created. Bill