SourceForge processor downloading wrong file

jelockwood commented 5 years ago

I used Recipe Robot 1.1.1 to create a recipe for downloading a project from SourceForge. According to the messages output by RR it successfully downloaded a real disk image as per the log below.

                      -----------------------------------
                     |  Welcome to Recipe Robot v1.1.1.  |
                      -----------------------------------
                                \   _[]_
                                 \  [oo]
                                   d-||-b
                                     ||
                                   _/  \_

Processing https://sourceforge.net/projects/smartmontools/ ...
Input path looks like a SourceForge URL.
    App name is: S.M.A.R.T. Monitoring Tools
Getting SourceForge project ID...
    SourceForge project ID is: 64297
Getting SourceForge description...
    SourceForge description is: Disk Inspection and Monitoring
Determining download URL from SourceForge RSS feed...
    Download URL is: https://sourceforge.net/projects/smartmontools/files/smartmontools/7.0/smartmontools-7.0-1.dmg
Downloading file for further inspection...
    9.73%
    19.45%
    29.18%
    38.91%
    48.63%
    58.36%
    68.09%
    77.81%
    87.54%
    97.27%
    Downloaded to /Users/itsupport/Library/Caches/Recipe Robot/2019-02-27_10-32-03_819896/smartmontools-7.0-1.dmg
Determining download format...
    File extension is dmg
Opening downloaded file...
    Successfully mounted disk image
Checking whether package is signed...
    Package is not signed
Expanding package to look for clues...
    Package expanded to: /Users/itsupport/Library/Caches/Recipe Robot/2019-02-27_10-32-03_819896/expanded
Trying to get bundle identifier from PackageInfo file...
    Bundle identifier (tentative): com.smartmontools.pkg
[WARNING] No apps found in payload.
Searching for existing AutoPkg recipes for S.M.A.R.T.+Monitoring+Tools...
    No results
Searching for existing AutoPkg recipes for S.M.A.R.T.MonitoringTools...
    No results
Searching for existing AutoPkg recipes for SMARTMonitoringTools...
    No results
[REMINDER] I can't tell whether to use CFBundleShortVersionString or CFBundleVersion for the version key of this app. Most apps use CFBundleShortVersionString, so that's what I'll use. You may want to verify that and modify the recipes if necessary.
Generating munki recipe...
[WARNING] I don't have enough information to create a PNG icon for this app.
    /Users/itsupport/Library/AutoPkg/Recipe Robot Output/S.M.A.R.T. Monitoring Tools/S.M.A.R.T. Monitoring Tools.munki.recipe
Generating jss recipe...
[REMINDER] Remember to manually set the category in the jss recipe. I've set it to "Productivity" by default.
[REMINDER] Make sure to keep S.M.A.R.T. Monitoring Tools.png with the jss recipe so JSSImporter can use it.
[WARNING] I don't have enough information to create a PNG icon for this app.
    /Users/itsupport/Library/AutoPkg/Recipe Robot Output/S.M.A.R.T. Monitoring Tools/S.M.A.R.T. Monitoring Tools.jss.recipe
Generating download recipe...
    /Users/itsupport/Library/AutoPkg/Recipe Robot Output/S.M.A.R.T. Monitoring Tools/S.M.A.R.T. Monitoring Tools.download.recipe

You've now created 25 recipes with Recipe Robot. Amazing.

I added the download and munki recipes to my repo, updated my repo in AutoPkg, created a matching override and tried running this.

Unfortunately I then got an error as follows.

Error running recipes

Use of undefined key in variable substitution: u'version'

mounting /Users/itsupport/Library/AutoPkg/Cache/local.munki. Monitoring Tools.T.R.A.M.S/downloads/%NAME%-%version%.dmg failed: hdiutil: attach failed - image not recognized

Further examination clearly shows what is going wrong is that using this recipe AutoPkg is actually downloading the MD5 file and not the 'real' disk image. An MD5 of course cannot be mounted as a disk image and hence the recipe fails as above.

The relevant section of the autopkg reciept file is as follows.

            <key>pkginfo</key>
            <dict>
                <key>catalogs</key>
                <array>
                    <string>testing</string>
                </array>
                <key>category</key>
                <string>Utilities</string>
                <key>description</key>
                <string>Disk Inspection and Monitoring</string>
                <key>developer</key>
                <string>smartmontools</string>
                <key>display_name</key>
                <string>S.M.A.R.T. Monitoring Tools</string>
                <key>name</key>
                <string>S.M.A.R.T. Monitoring Tools</string>
                <key>unattended_install</key>
                <true/>
            </dict>
            <key>verbose</key>
            <integer>1</integer>
        </dict>
    </dict>
    <dict>
        <key>Input</key>
        <dict>
            <key>SOURCEFORGE_FILE_PATTERN</key>
            <string>\.dmg</string>
            <key>SOURCEFORGE_PROJECT_ID</key>
            <integer>64297</integer>
        </dict>
        <key>Output</key>
        <dict>
            <key>url</key>
            <string>https://sourceforge.net/projects/smartmontools/files/smartmontools/7.0/smartmontools-7.0-1.dmg.md5/download</string>
        </dict>
        <key>Processor</key>
        <string>com.github.jessepeterson.munki.GrandPerspective/SourceForgeURLProvider</string>
    </dict>
    <dict>
        <key>Input</key>
        <dict>
            <key>filename</key>
            <string>%NAME%-%version%.dmg</string>
            <key>url</key>
            <string>https://sourceforge.net/projects/smartmontools/files/smartmontools/7.0/smartmontools-7.0-1.dmg.md5/download</string>
        </dict>
        <key>Output</key>
        <dict>
            <key>download_changed</key>
            <true/>
            <key>etag</key>
            <string>"5c28e094-3a"</string>
            <key>last_modified</key>
            <string>Sun, 30 Dec 2018 15:13:24 GMT</string>
            <key>pathname</key>
            <string>/Users/itsupport/Library/AutoPkg/Cache/local.munki. Monitoring Tools.T.R.A.M.S/downloads/%NAME%-%version%.dmg</string>
            <key>url_downloader_summary_result</key>
            <dict>
                <key>data</key>
                <dict>
                    <key>download_path</key>
                    <string>/Users/itsupport/Library/AutoPkg/Cache/local.munki. Monitoring Tools.T.R.A.M.S/downloads/%NAME%-%version%.dmg</string>
                </dict>
                <key>summary_text</key>
                <string>The following new items were downloaded:</string>
            </dict>
        </dict>
        <key>Processor</key>
        <string>URLDownloader</string>
    </dict>
    <dict>
        <key>Input</key>
        <dict/>
        <key>Output</key>
        <dict/>
        <key>Processor</key>
        <string>EndOfCheckPhase</string>
    </dict>
    <dict>
        <key>RecipeError</key>
        <string>Error in local.munki. Monitoring Tools.T.R.A.M.S: Processor: AppDmgVersioner: Error: mounting /Users/itsupport/Library/AutoPkg/Cache/local.munki. Monitoring Tools.T.R.A.M.S/downloads/%NAME%-%version%.dmg failed: hdiutil: attach failed - image not recognized</string>
    </dict>
</array>
</plist>

The recipe in question is available here - https://github.com/jelockwood/recipes/blob/master/S.M.A.R.T.%20Monitoring%20Tools.download.recipe

homebysix commented 5 years ago

Good catch! Could you try changing a section of your download recipe from this:

            <key>SOURCEFORGE_FILE_PATTERN</key>
            <string>\.dmg</string>

to this:

            <key>SOURCEFORGE_FILE_PATTERN</key>
            <string>\.dmg$</string>

and let me know whether that works?

jelockwood commented 5 years ago

It fails differently now, it comes back with the following.

    <dict>
        <key>RecipeError</key>
        <string>Error in local.munki. Monitoring Tools.T.R.A.M.S: Processor: com.github.jessepeterson.munki.GrandPerspective/SourceForgeURLProvider: Error: nothing to repeat</string>
    </dict>

homebysix commented 5 years ago

I've done a bit of work on this, and found two main issues:

The file regex was wrong because SourceForge downloads are all suffixed with /download. Adjusting the SOURCEFORGE_FILE_PATTERN to \.dmg\/download$ appears to resolve that issue reliably (8835819).
This specific SourceForge project is not signed and the pkg doesn't contain any apps (see screenshot below), so Recipe Robot can't find any version information. This should be a pretty rare case, but just to be safe I added some warnings to the output for these situations (308aa95). It's likely you'll need to make some manual modifications to the recipes produced by Recipe Robot, including but not limited to:
- Adding some kind of Versioner processor (possibly this one) into the download recipe (and potentially FlatPkgUnpacker to get to the actual binary file).
- Replacing AppPkgCreator in the pkg recipe with PkgCopier, in order to grab the installer from the downloaded disk image.
- Switching the parent on the munki recipe to pkg instead of download, and adjusting pkginfo to include considerations like blocking_applications.
- Removing the AppDmgVersioner processor from the Munki recipe.

I'm not going to spend more time on this particular use case, because the SMART Monitoring Tools is quite far from Recipe Robot's target scope of basic Mac apps. But I hope this gives you a head start on those recipes. Thanks!

jelockwood commented 5 years ago

@homebysix Many thanks for your looking in to this and your suggestions. I believe I have now managed to get a working recipe together which even manages to produce a sensible version number. It seems without the auto-generated processor step for DMGVersioner the munki recipe falls back to examining the installer package and finds inside that the PackageInfo file and from that gets a version number.

Your regex for the SourceForge download itself works perfectly.

If your interested you can see my (working) recipes at https://github.com/jelockwood/recipes and we are of course talking about the S.M.A.R.T Monitoring ones.

homebysix / recipe-robot

SourceForge processor downloading wrong file #144