search

gnudatalanguage / gdl

GDL - GNU Data Language
GNU General Public License v2.0
277 stars 61 forks source link

sometimes get % SHMMAP: Existing file too short for desired mapping. #1853

Closed brandy125 closed 1 month ago

brandy125 commented 4 months ago

Sometimes I get the following error message in my program:

% SHMMAP: Existing file too short for desired mapping.

When starting the same program again it works. Unfortunately I have no short example here to reproduce the error but let me ask if this is a known issue or what can I do to avoid this? Since shmmap is available now in GDL I use it very often so that many CPUs can work in parallel on the same array and it works perfectly beside this error message that comes from time to time. Should I "catch" it and execute the command again or what do you guys suggest?

GillesDuvert commented 4 months ago

I got this message when doing tests. It means that the file is (now?) not long enough in bytes to enable the requested number of values. E.g. the file was created as 1000 chars and you want to map 300 4-byte ints = 1200 bytes. If that's not the case, it may be a problem of concurrent access I suppose.

brandy125 commented 4 months ago

One program creates the array with a "fixed" size and the other wants to map it with that known size. Seems that it is somehow a timing problem because a simple wait resolves the problem. I think it should be possible to isolate the problem in a separate program.

GillesDuvert commented 4 months ago

Certainly a concurrency problem. All this was programmed without any attempt at being threadsafe. I'm sure you can provide a simple example ?

GillesDuvert commented 4 months ago

@brandy125 changes in #1865 may help.

I need to know if using the SYSV version of shmmap is better or worse. To use it, you have to create the shared area with option /SYSV, GET_OS_HANDLE=name and then use /SYSV, OS_HANDLE=name when mapping the same are in the 'client' program.

The (default) POSIX mmap() scheme uses a file, contrary to the SYSV scheme. I would guess reading/writing to a memory-mapped file will use the inherent lock access to files to prevent concurrency problems, but apparently not.

ideally I'll need a simple example of what you do to test on a 8 threads old machine.

brandy125 commented 4 months ago

@GillesDuvert I am testing extensively the latest version with SYSV and GET_OS_HANDLE and so far it is running like a charm. It like this version that allows me to work with handles, nicely increasing numbers that I pass to the clients. Now I know that the programs will work exactly on the shared area that was specified for them. Before I had situations where a client program did not die and worked in parallel with the new program on the same area.

So far so good. I will keep you updated...

GillesDuvert commented 4 months ago

excellent, I keep fingers crossed.