PhysicsFS 3.0.0 support

[jdolan]

Ryan Gordon has spent a development cycle on PhysicsFS, and is planning to release 3.0.0 soon. It's available for testing as 2.1.1, or from Mercurial HEAD. Here's an excerpt of the beta announcement:

( This work was funded by my Patreon! If you like me working on things like PhysicsFS and other useful pieces of code, consider throwing some money in the pot: https://patreon.com/icculus )

So I knuckled down and pushed hard to get the PhysicsFS development branch where I want it for a major-version release.

Now it's ready to go beta!

PhysicsFS 2.1.1, which will stabilize into 3.0.0, is ready for you to test. It should be safe for anyone on any official release of PhysicsFS to update: the API has been expanded over time, and functions have been deprecated, but we are still source and binary compatible with programs written against PhysicsFS 1.0.0. In theory, you can drop in a new build and it'll Just Work.

Please try the latest in revision control with your apps and games and report issues. A lot of code changed pretty quickly, so it's possible obvious things are broken (and easy to fix), so feedback is extremely useful now.

WHAT'S NEW:

It's hard to quantify everything that changed, because an enormous amount of reworking went on behind the scenes, files got renamed a bunch, etc. If you want specifics, pull the latest from Mercurial and run "hg log -b default -r 5cabc31eb563:3396e6dd19fb" to get every commit involved, but I'll summarize the best things here:

• PhysicsFS now is super-easy to build. You can usually just drop all the C files into your project and compile it with everything else without any magic configuration step. The CMake file is still there and useful for packaging, etc, but it's 100% optional.

• ZIP files can now use the newer zip64 format.

• ZIP files may be password-protected. As the PkWare specs specify, each file in the .zip may have a different password, so you call PHYSFS_openRead(a, "file_that_i_want.txt$PASSWORD") to make it work. Note that this is a wildly insecure way to protect your app's data, both in how you'd have to manage passwords and that "traditional" PkWare crypto is not really hardened anyhow. But if you have a basic password-protected archive, PhysicsFS can get into it now!

• 7zip support has been rewritten and improved.

• ISO9660 archives are now supported.

• VDF (Gothic/Gothic2) archives are now supported.

• SLB (Independence War) archives are now supported.

• Everything behind the scenes now uses an abstract i/o interface (PHYSFS_Io) instead of talking directly to files, and this interface is available to applications. This allows you to mount anything as an archive that you can wrap in a PHYSFS_Io, through the new PHYSFS_mountIo() function. The obvious and most useful implementations are already implemented for you behind the scenes on top of PHYSFS_Io: PHYSFS_mountMemory() to mount an archive that's in a block of RAM, and PHYSFS_mountHandle() to mount from a PHYSFS_File*...that is to say: you now have an interface to do archives inside archives.

• The abstract interface that PhysicsFS uses to talk to archives (PHYSFS_Archiver) is now public, so apps can implement their own archivers and register them into the system. If you have a custom archive format for your app, you can plug it into PhysicsFS at runtime without modifying PhysicsFS itself.

• There's now a PHYSFS_getPrefDir() to figure out where it is safe to write files on a given platform. This is usually a per-user, per-application space, and should be used instead of PHYSFS_getUserDir(). It might report something under ~/Library/Application Support on a Mac, somewhere under AppData for Windows, or ~/.local/share on Linux, etc.

• There's now a PHYSFS_unmount() to match 2.0.0's PHYSFS_mount().

• There's now a PHYSFS_utf8FromUtf16(), so stop using PHYSFS_utf8FromUcs2(). :)

• There's now PHYSFS_utf8stricmp() for case-insensitive UTF-8 string comparison, in case you need to compare filenames.

• Enumerating files can now report errors, instead of quietly dropping files when there were problems. Enumeration callbacks can now return results too ("keep going", "stop enumerating, no error I just got what I needed", "something went wrong, stop and return an error to the app").

• There is now a PHYSFS_stat() call that returns metadata on a specific file as a whole instead of using PHYSFS_isDirectory(), PHYSFS_getLastModTime(), etc separately).

• There's now a PHYSFS_readBytes() function that operates more like Unix read()...the older PHYSFS_read() operates more like ANSI C's fread(), but had undefined behavior if it reads half an object. There is also an equivalent PHYSFS_writeBytes().

• Errors are now reported by numeric codes your app can process (there's a function to turn them into human readable strings). Apps can set the per-thread error code now, too, which is useful if you're writing an PHYSFS_Archiver or PHYSFS_Io.

• The OS/2 port has Unicode support now.

• The Windows port now uses UTF-16 for Unicode instead of UCS-2, and dropped the ANSI fallbacks (so among other hurdles, you'll need to bring your own UNICOWS.DLL if you want Win95 support still).

• Lots of improvements, redesigns, bug fixes, corner cases, and optimizations.

• There's now a buildbot that makes sure this builds across two dozen different targets on every commit: https://physfs-buildbot.icculus.org/waterfall

• New platforms: iOS, Android, Emscripten, WinRT (UWP, Windows Phone, Windows Store), Win64, QNX, Solaris, Hurd, Debian/kFreeBSD, ArcaOS, probably others.

• Dropped platforms: BeOS (Haiku is still supported), Windows CE (Windows Phone is supported), Windows 95/98/ME (WinXP and later are supported) MacOS Classic (macOS and iOS are supported). Even these might work with minor patches if there's an urgent demand.

• Probably many other things.

That's all: please enjoy! And please test and report back!

Thanks, --ryan.

We should probably try to get ahead of this, and cut a feature branch to test this out on all 3 platforms. It sounds like a few bugs / issues we've had with PhysicsFS (pak file enumeration, zip file rewinding, etc..) are likely fixed.

[jdolan]

cc @Paril @kaadmy

[kaadmy]

Shouldn't this mostly be a drop-in test?

[jdolan]

Mostly. But there are probably a few new toys we can take advantage of / remove some of our own code. So with that in mind, and because 3.0.0 won't be officially released for a little while, I think it warrants a feature branch. Our early feedback would help Ryan out, too.

[jdolan]

Also, just something to consider, I would entertain the idea of bundling PhysicsFS in deps like we do for Minizip, given Ryan's claim of it compiling universally without any fuss. I'm not saying that's the right path, but it's something we could look at. I know that, in the past, it's been a challenge finding the latest stable release on all 3 platforms.

[Paril]

@jdolan can we confirm that this fixes the pak search problem?

[jdolan]

The fix that Ryan proposed we try is in, yes. I have not yet tested it.

[Paril]

I will try it in a branch. Will put it in deps too.

[jdolan]

This has been merged in #496. However, we're currently straddling the line between PhysFS 2.x and PhysFS 3.x (we're compatible with both). This is to accommodate OS X (MacPorts still not shipping PhysFS 3) and our Jenkins build slaves (Ubuntu 16.04 stuck at 2.x, Fedora 24 stuck at 2.x for MinGW as well).

Maybe in a few months, we can update fully to PhysFS 3. But, this is in now, and folks looking to build Quetoo with it can do so.