get rid of tempfile (and mkstemp) dependency

[sezero]

I don't know how doable this is. The depackers side mostly (if not wholely?) moved to in-memory depacking, but the whole prowizard suite still relies on this functionality.

[AliceLR]

moved to in-memory depacking

This should be considered temporary at best—this is almost certainly going to break depacking on low RAM systems and libxmp should be able to fall back to temporary files (maybe with a xmp_set_player flag to prevent it or to enable this fallback).

[sezero]

going to break depacking on low RAM systems

and there is that, too..

and libxmp should be able to fall back to temporary files (maybe with a xmp_set_player flag to prevent it or to enable this fallback).

Or maybe a configuration option

[AliceLR]

Or maybe a configuration option

That seems reasonable—I think whether or not the target would be low enough RAM for that to be useful would be known ahead of time. I don't think adding that kind of fallback is particularly urgent, anyway, and could wait until someone complains.

Re: ProWizard, I think it used to unpack to RAM at one point, but I'm not familiar enough. Getting ProWizard to the point where it could output to RAM or RAM+stdio would be a big project. There is output size counting deadcode in many of its unpackers that would be ideal to restore for this (to reserve a buffer up front and/or quickly fall back to stdio). The rest could probably be abstracted by adding pw_write to replace fwrite.

[ccawley2011]

With temporary files, there is still a potential portability issue where embedded platforms may require more platform-specific code to determine where temporary files should be stored, or may not have enough writable storage to create the file to begin with.

Another possibility is to allow depackers to provide custom HIO streams, similarly to how gzopen works in zlib. This would definitely be the most difficult approach, though, as it would require depackers to support partial decompression, and may be slower for loaders that seek backwards often.

For comparison, how much RAM does libxmp usually require for module loading and playback?

[AliceLR]

With temporary files, there is still a potential portability issue where embedded platforms may require more platform-specific code to determine where temporary files should be stored, or may not have enough writable storage to create the file to begin with.

Another possibility is to allow depackers to provide custom HIO streams, similarly to how gzopen works in zlib. This would definitely be the most difficult approach, though, as it would require depackers to support partial decompression, and may be slower for loaders that seek backwards often.

That's a reasonable point—I was thinking of embedded systems with SDKs that already provide tempfile functions, but those are probably a minority. libxmp already has handling for custom temporary file directories and a mkstemp fallback, but making tempfiles a configuration option like sezero suggested tied to Autoconf detection of mkstemp (or Win32, which has mkstemp.c) could help.

Due to the complexity of the loader test functions and formats with arbitrary seeking (including S3M and IT!), I think pushing off the responsibility of file IO onto the user might work better than trying to stream to the loaders. This might only work with xmp_load_module_from_callbacks and would require several extra callbacks, which can't be added without breaking API changes. (xmp_callbacks already needs a read open callback to support things like MOD/STM/MED2 songs and Startrekker AM configuration files, so adding a tempfile open callback and write callback wouldn't be too out of the question.)

For comparison, how much RAM does libxmp usually require for module loading and playback?

I haven't really instrumented this, but a rough overview from my understanding (theoretically assuming depacking works with xmp_load_module_from_memory and xmp_load_module_from_callbacks here, which it doesn't yet):

1. User allocates initial file to RAM if using xmp_load_module_from_memory, or xmp_load_module_from_callbacks with RAM.
2. xmp_load_module_*
   1. Depacking.
      1. Positive packer/archive detection: a new buffer is allocated for the uncompressed size or dynamically expanded to the uncompressed size.
      2. stdio and xmp_load_module_from_callbacks: the archive handle is closed on success. For callbacks, this is only if close_func is provided. close_func can be used as a way to guarantee the initial RAM is freed before proper loading starts, which leaves the loading process at the same RAM overhead as if xmp_load_module_from_memory was used with an uncompressed module. If this is xmp_load_module_from_memory, both the compressed and uncompressed buffers will now be consuming RAM.
   2. Module testing and loading.
      1. Patterns, tracks, instruments, subinstruments, samples, sample data, module extras, instrument extras, extra sample data are allocated here. These usually work out to larger than the input file due to the size of track/event data, unpacked/decoded samples, and instrument structs.
      2. Occasionally there are temporary buffers involved which IIRC cap out in the 256kB range (or up to 256MB for packed samples). I've been changing some loaders to keep these around through multiple pattern/sample loads, which may increase RAM overhead to varying degrees for the affected loaders. I don't think this increase is particularly serious though. These are usually released after pattern or sample loading is complete.
   3. libxmp_prepare_scan and libxmp_scan_sequences allocate and shrink some buffers that are fairly irrelevant in size to everything else (~5k max with current module length limitations I think).
   4. Depacker HIO_HANDLE handles and buffers are released here.
3. User frees handles that they manage here. If user didn't provide a close_func to xmp_load_module_from_callbacks, user frees/closes handle here.
4. xmp_start_player allocates the channel_data (up to ~100kB worst case for IT) and mixer_voice (up to 20kB) arrays.

With a packed module in the MB size range you'd have to keep the packed file (with the memory and callbacks loaders only, theoretically enabled for depacking), the unpacked file, and all data allocated by the loader in RAM at the same time (which in the case of IT or OggMod XMs, could blow up from sample decoding). I think it would require a fairly low RAM system (<=16MB) to be an issue, but this includes several old/embedded systems libxmp can or may plausibly run on.