Using packagesutil to Create a Project and Package From Scratch

[TrevorChan1]

Hello,

I'm currently trying to create Mac packages and installers for multiple pieces of software, so I'm trying to help automate the process by only creating the .pkg files using packagesutil and packagesbuild commands in scripts. For packagesutil, I was wondering how you can create a .pkgproj file from scratch that can then be altered using the packagesutil set commands. I tried creating a file with the .pkgproj extension and edit that, but when I tried to open it on Packages to check the changes, it said "The document "testPkgUtils.pkgproj" could not be opened. The data is not in the correct format." Is there a certain way I need to make the file (without using the GUI) so I can edit it?

Also, when using the packagesutil commands, I was wondering how you can set the Payload of a package using only the command line.

Thank you in advance for your help!

[TrevorChan1]

I found the test script qa_test.sh and I've been fiddling around trying to understand it. I've gotten the code to run, but there are a few errors. On top of my previous questions, I was wondering why I'm getting these errors:

Simple commands like: packagesutil get project name --file ../../tests/test.distribution.pkgproj give me errors like the one below for any command involving distribution.pkgproj files, but get project name works with the rat pkgproj files. packagesutil: /Users/trevorchan/Downloads/packages/packagesutil/tests/test.distribution.pkgproj: Invalid type for value: PROJECT_SETTINGS/ADVANCED_OPTIONS

Also, I get these build errors:

2022-07-05 12:48:06.959 xcodebuild[3413:40510] Requested but did not find extension point with identifier Xcode.IDEKit.ExtensionSentinelHostApplications for extension Xcode.DebuggerFoundation.AppExtensionHosts.watchOS of plug-in com.apple.dt.IDEWatchSupportCore
2022-07-05 12:48:06.960 xcodebuild[3413:40510] Requested but did not find extension point with identifier Xcode.IDEKit.ExtensionPointIdentifierToBundleIdentifier for extension Xcode.DebuggerFoundation.AppExtensionToBundleIdentifierMap.watchOS of plug-in com.apple.dt.IDEWatchSupportCore
--- xcodebuild: WARNING: Using the first of multiple matching destinations:
{ platform:macOS, arch:x86_64, id:564DEC2C-19AA-F48D-7F78-F3FF57D64DA9 }
{ platform:macOS, name:Any Mac }
--- xcodebuild: WARNING: Using the first of multiple matching destinations:
{ platform:macOS, arch:x86_64, id:564DEC2C-19AA-F48D-7F78-F3FF57D64DA9 }
{ platform:macOS, name:Any Mac }

** BUILD FAILED **

The following build commands failed:

    CompileC /Users/trevorchan/Downloads/packages/packagesutil/build/Release/Build/Intermediates.noindex/packagesutil.build/Release/packagesutil.build/Objects-normal/x86_64/PKGArchive.o /Users/trevorchan/Downloads/packages/archive_utilities/PKGArchive.m normal x86_64 objective-c com.apple.compilers.llvm.clang.1_0.compiler (in target 'packagesutil' from project 'packagesutil')

(1 failure)

[packagesdev]

1. You can't populate the payload using packagesutil.
2. The .pkg proj files are PropertyList files. When you create a new project in Packages, it uses one of the templates located in /Library/Application Support/fr.whitebox.packages/Projects Templates.
3. These template files can not be used as-is as they include MACROs that will be resolved at the end of the creation of the new project based on the name of the project and default values (the ones that can be customized in the Preferences > Templates pane). Additionally, some of the sections of the files are empty by default (e.g. the payload hierarchy). It gets populated either when you visit the Payload pane for the first time or when the package is built.
4. Depending on how the much the multiple pieces of software in your case look like, you could edit the Property List and in this case your best bet is to create a project, add a file to the payload and then you can figure out the format used in the Property List for system folders, file items. Or you could use the same project for similar pieces and play with the user defined settings to customize the build. Undocumented feature: it's possible to use an user defined setting for the source path of an item in the payload.

Ref. http://s.sudre.free.fr/Software/documentation/Packages/en_2017/Project_Configuration.html#9

[packagesdev]

Compilation issue: it may depend on the Xcode version and SDK. Is the reason for the failure listed?

[TrevorChan1]

Thank you for your quick response!

I have a few questions:

1. From what I understand, the two main things that can't be automated using the command line are creating the .pkgproj file and adding the payload. For automating creating the .pkgproj file, would there be any errors caused by me just copying the .pkgproj file inside of one of the templates and then using packagesutil to edit things like the name, package settings, etc.?
2. For the Property List for system folders you mentioned, do you mean that I could automate creating the Payload by seeing what changes in the code itself and then just manually change what I want before building the project? Also, how can I use user-defined settings to set source paths for the payload?
3. How does the certificate-keychain and identity work? Are these equivalent to signing the packages or would I need to have Apple individually sign each package and the .pkg installer for the installer to be ready for distribution?

For the compilation error, it mentioned that the build command CompileC failed with PKGArchive.o PKGArchive.m normal x86_64 objective-c com.apple.compilers.llvm.clang.1_0.compiler (in target 'packagesutil' from project 'packagesutil')

The full error message is the 2nd to last line in the 2nd comment

Thank you!

[TrevorChan1]

Also, for the User Defined Settings, is it possible to make one setting based on the value of another? For example, since I'm currently using a template .pkgproj as the baseline project, I can't change the title of the installer. Is there a way for me to set the Title setting in User Defined Settings to the value of Distribution Name?

[TrevorChan1]

Update: I've been able to manually write the payload code into the .pkgproj file, but I don't think this is a very efficient method for automation for if there are a lot of files or if there are different permissions settings on different files. Are there any other possible methods of generating these file path blocks of code or writing them using the command line?

[packagesdev]

Usually, even when there are a lot of files, you only end up adding a few references in the Payload hierarchy because you just need to add the .app bundle folder or the parent folder. If you are manually building the entire payload hierarchy, you might be able to just add the root folder of your payload hierarchy.

[TrevorChan1]

Hello,

Thank you! Opening payload on the template file I'm using made it a lot easier by initializing the hierarchy so I only had to add a few folders like you mentioned. I've been able to automate the process of creating the payload now, but I'm having an issue with building the package file. I've been able to build using packagesbuild on different projects and the terminal itself isn't running into any errors, but packagesbuild is taking a lot of time (over 20 minutes) and I can't tell if it's frozen or not.

What could be causing build to freeze like this? Running verbose, it shows that it stops at "Payload" and doesn't get to "Assemble file hierarchy (done)". I'm able to open the project file with the GUI and it doesn't seem like any of the files are invalid.

(Also, the files it's connected to are from a shared externally mounted drive. Could this cause any errors?)

[packagesdev]

No idea. This would require to use Instruments or a similar tool to try to find where the slowness (or wait) is coming from.

[TrevorChan1]

I think the freezing was caused by just having a lot of files. I'm trying to zip the largest directories and just work with post-installation scripts for unzipping.

I am having a problem with including directories, though. When you add a directory to the payload through editing the XML rather than through the GUI, it seems like the installing user won't have access to subdirectories (it'll have that little red mark in the bottom right). Is there a way to tell the XML to look recursively into the directory or do I need to manually go through directories and add each individual item?

[packagesdev]

Is there a way to tell the XML to look recursively into the directory or do I need to manually go through directories and add each individual item?

You don't need to add the contents of the directory if all the items inside the directory should be in the payload.

Regarding the little red mark, you probably have the wrong owner or POSIX permissions set.

Typically a folder entry in the plist file would look like this:

<dict>
    <key>CHILDREN</key>
    <array/>
    <key>GID</key>
    <integer>80</integer>
    <key>PATH</key>
    <string>../mysource/folder</string>
    <key>PATH_TYPE</key>
    <integer>1</integer>
    <key>PERMISSIONS</key>
    <integer>493</integer>
    <key>TYPE</key>
    <integer>3</integer>
    <key>UID</key>
    <integer>0</integer>
</dict>

[TrevorChan1]

For the target file / directory to add to the Payload, my code generates:

<dict>
    <key>CHILDREN</key>
    <array/>
    <key>GID</key>
    <integer>0</integer>
    <key>PATH</key>
    <string>absolute/path/to/the/file</string>
    <key>PATH_TYPE</key>
    <integer>0</integer>
    <key>PERMISSIONS</key>
    <integer>493</integer>
    <key>TYPE</key>
    <integer>3</integer>
    <key>UID</key>
    <integer>0</integer>
</dict>

I think the only differences are in my GID and path_type (Although if I'm remembering right I'm using path_type 0 for absolute file paths). Would these affect the accessibility at all / the speed it takes for the build to run? It does seem like expanding in the project file shows that the Permissions are different than the permissions I give to the target directory. Is there a way to make subdirectories and files in the target directory inherit its permissions settings?

[packagesdev]

Would these affect the accessibility at all / the speed it takes for the build to run?

Probably not.

Is there a way to make subdirectories and files in the target directory inherit its permissions settings?

Just don't include the subdirectories and files in the payload. They should be added automatically.

[TrevorChan1]

Hello,

I've been fiddling with it and it seems like a big part of why it's taking so long is because I'm working with files located on an externally mounted drive. Do you think that this could be messing with the permissions of the subdirectories?

When I try to just add a directory to the payload it's able to grab it and the files inside of it just fine, but for some reason subdirectories and the files in them only have rwx------ permissions, even though both the source directory I'm adding to the payload and the subdirectories/files themselves have rwxr-xr-x permissions. What could be changing the accessibility permissions?

[packagesdev]

Do you think that this could be messing with the permissions of the subdirectories?

Not with the POSIX permissions. And the ownership should be inherited from the parent item in the Packages payload hierarchy.

"What could be changing the accessibility permissions?"

The rwx------ permissions is what you get in the payload or when you install the package?

[TrevorChan1]

The original files have rwxr-xr-x then after adding the payload giving the directories permission 493 the files show they have rwx------ permissions on the built installer and inside of the GUI payload before building too. Since you mentioned they inherit from POSIX permissions could it be because I'm getting the files from an externally mounted Windows drive?

[TrevorChan1]

Finally figured out the permissions issue was with the filing system (when loaded it would automatically set the files to rwx------ which was why it couldn't grab directories recursively). Thank you so much for all of your help!