Python exception on checking software

[acodega]

Since there have been a lot of updates, I've revisited using AutoPkgr and my cloud hosted JSS.

I get this error when checking for new software, my JSS account is properly configured and the DP is being mounted. Same result if I run AutoPkgr from the DP or from a different machine.

12/1/14 11:27:30.422 AM AutoPkgr[30631]: (2/2) Processing Firefox.jss...
12/1/14 11:28:49.764 AM AutoPkgr[30631]: (FULL AUTOPKG TRACEBACK) 
  File "/usr/local/bin/autopkg", line 1334, in <module>
    sys.exit(main(sys.argv))
  File "/usr/local/bin/autopkg", line 1328, in main
    exit(subcommands[verb]['function'](argv))
  File "/usr/local/bin/autopkg", line 1152, in run_recipes
    autopackager.process(recipe)
  File "/Library/AutoPkg/autopkglib/__init__.py", line 466, in process
    self.env = processor.process()
  File "/Library/AutoPkg/autopkglib/__init__.py", line 295, in process
    self.main()
  File "/Library/AutoPkg/autopkglib/JSSImporter.py", line 634, in main
    ssl_verify=sslVerify, repo_prefs=repos)
  File "/Library/Python/2.7/site-packages/python_jss-0.4.2-py2.7.egg/jss/jss.py", line 169, in __init__
    self.distribution_points = distribution_points.DistributionPoints(self)
  File "/Library/Python/2.7/site-packages/python_jss-0.4.2-py2.7.egg/jss/distribution_points.py", line 85, in __init__
    self.dp_info = self.jss.DistributionPoint().retrieve_all()
  File "/Library/Python/2.7/site-packages/python_jss-0.4.2-py2.7.egg/jss/jss.py", line 317, in DistributionPoint
    return self.factory.get_object(DistributionPoint, data)
  File "/Library/Python/2.7/site-packages/python_jss-0.4.2-py2.7.egg/jss/jss.py", line 454, in get_object
    result = self.jss.get(url)
  File "/Library/Python/2.7/site-packages/python_jss-0.4.2-py2.7.egg/jss/jss.py", line 202, in get
    self._error_handler(JSSGetError, response)
  File "/Library/Python/2.7/site-packages/python_jss-0.4.2-py2.7.egg/jss/jss.py", line 189, in _error_handler
    raise exception
jss.jss.JSSGetError: JSS Error. Response Code: 404  Response" 
12/1/14 11:28:49.765 AM AutoPkgr[30631]: [DEBUG] Error [1] Error running recipes 
 A Python exception occurred during the execution of autopkg, see the console log for more details.

[ERROR] jss.jss.JSSGetError: JSS Error. Response Code: 404  Response" 
12/1/14 11:28:49.850 AM AutoPkgr[30631]: [DEBUG] Nothing new was downloaded.
12/1/14 11:28:49.850 AM AutoPkgr[30631]: [DEBUG] Completed AutoPkg Task:
  /usr/bin/python /usr/local/bin/autopkg run --recipe-list /Users/macadmin/Library/Application Support/AutoPkgr/recipe_list.txt --report-plist /var/folders/b3/jxplz2091hz9qb16wg962_4c0000gn/T/DE1D16CF-06E8-4669-B6A7-036D1C986DB0-30631-00052D8FF085E367

[eahrold]

So it looks like you're exactly back to this...

https://github.com/lindegroup/autopkgr/issues/170#issuecomment-61481196

Is this the same MM that was hosed? Were you ever able to figure out what was different after your reinstall?

[acodega]

It is the same rebuilt Mac mini. I've since updated to the new version of JSS AutoPkg Addon and AutoPkgr.

[eahrold]

This is crazy. From the previous issue it looked as thought you started getting past all that once you wiped the MM, and were just having errors connecting to the DP. Now you're back raising at the beginning of the processor.

I doubt it will help, but you could try removing all auto-added, DP's and manually add them back with the "Add" panel (new with 1.1.2 & python-jss 0.4.0)

Also I don't see me asking this on the other thread, double check that this didn't get mis-set somewhere along the way.

defaults read com.github.autopkg JSS_URL

[acodega]

defaults read com.github.autopkg JSS_URL

reports the correct URL. Manually adding DPs doesn't help. I tried using my Amazon Distribution Point, as if it was a JDS server, and that doesn't work either. I'm unsure if this is happening before trying to mount the DP or after.

[eahrold]

Hey Adam, At this point I think we should see if python-jss is working at all with your setup

cd /Library/Python/2.7/site-packages
python

# Now in python interpreter

>>> import jss

# Make a JSS object to test the DP:

>>> j = jss.JSS(url='https://jss.jamfcloud.com/yourinstance', user='privilegedusername', password='password')

>>> j.DistributionPoint()

# Should return the available DP's on your JSS (no tracebacks...)

[acodega]

It looks to be working.

(For others who may be following along, there should be a closing parens at the end of JSS(url=..)

This is what printed:

>>> j.DistributionPoint()
--------------------------------------------------
List index:     0
id:     1
name:       myDP
--------------------------------------------------

>>>

[eahrold]

Do this again now with

>>> j.DistributionPoint().retrieve_all()

(basically I'm trying to follow the traceback and see if you can make it hanging)

[eahrold]

@acodega, I've got something even more indicative, that will help diagnose the preference file too,

# in bash terminal
export JSS_URL=$(defaults read com.github.autopkg JSS_URL)
export JSS_USERNAME=$(defaults read com.github.autopkg API_USERNAME)
export JSS_PASSWORD=$(defaults read com.github.autopkg API_PASSWORD)

python

# Now in python interpreter

>>> import os, jss

>>> j = jss.JSS(url=os.environ.get("JSS_URL"), user=os.environ.get("JSS_USERNAME"), password=os.environ.get("JSS_PASSWORD"))

>>> j.DistributionPoint()
>>> j.DistributionPoint().retrieve_all()

Since the other way worked, I'm hoping this will raise an exception, so at least we have something to go on.

[acodega]

Okay I'm following you. Wasn't able to sit down with this today but will tomorrow. Thanks!

[acodega]

Ahh here we go

>>> j.DistributionPoint()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Python/2.7/site-packages/python_jss-0.4.2-py2.7.egg/jss/jss.py", line 317, in DistributionPoint
    return self.factory.get_object(DistributionPoint, data)
  File "/Library/Python/2.7/site-packages/python_jss-0.4.2-py2.7.egg/jss/jss.py", line 454, in get_object
    result = self.jss.get(url)
  File "/Library/Python/2.7/site-packages/python_jss-0.4.2-py2.7.egg/jss/jss.py", line 202, in get
    self._error_handler(JSSGetError, response)
  File "/Library/Python/2.7/site-packages/python_jss-0.4.2-py2.7.egg/jss/jss.py", line 189, in _error_handler
    raise exception
jss.jss.JSSGetError: JSS Error. Response Code: 404  Response" 

So there's a problem with how it reads the variable?

[eahrold]

I don't think your JSS_URL is set correct in the com.github.autopkg preference

After setting the env what do you get when you do

echo $JSS_URL

It should be identical to the one you did on the initial python-jss test

[acodega]

That variable is correct though, https://jss.jamfcloud.com/[instance]/

https://[instance].jamfcloud.com/ doesn't work in AutoPkgr.

[acodega]

To clarify, if you enter https://[instance].jamfcloud.com/ it changes to https://jss.jamfcloud.com/[instance]/

[eahrold]

Correct we've started handling url redirection in AutoPkgr to address JAMF's two different styles of instances (old and new). So if you enter https://jss.jamfcloud.com/[instance] and it sends a redirect to https://[instance].jamfcloud.com then AutoPkgr sets that as the URL to use and updates the UI accordingly (new style).

And conversely if you set https://[instance].jamfcloud.com and get a redirect to https://jss.jamfcloud.com/[instance] it will set that as the URL to use (older style).

That being said It makes no sense how you successfully got the DP above when manually entering the url, but it wouldn't accept the defaults value.

The other thing to try is to make sure it makes it into the python environment.

so do all the export with defaults, then go into the python environment and

>>> import os,jss
>>> print os.environ.get('JSS_URL')

When you did it manually which did you use?

[acodega]

It get's the correct URL:

>>> print os.environ.get('JSS_URL')
https://jss.jamfcloud.com/[myinstance]/
>>>

I always use the "regular" URL, https://jss.jamfcloud.com/[myinstance]/

[eahrold]

And when you copy and paste that value into

j = jss.JSS(url='https://jss.jamfcloud.com/yourinstance', user='privilegedusername', password='password')

It works?

[acodega]

I think this may be more helpful

>>> j = jss.JSS(url=os.environ.get("https://jss.jamfcloud.com/[instance]/"), user=os.environ.get("JSS_USERNAME"), password=os.environ.get("JSS_PASSWORD"))
>>> j.DistributionPoint()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Python/2.7/site-packages/python_jss-0.4.2-py2.7.egg/jss/jss.py", line 317, in DistributionPoint
    return self.factory.get_object(DistributionPoint, data)
  File "/Library/Python/2.7/site-packages/python_jss-0.4.2-py2.7.egg/jss/jss.py", line 454, in get_object
    result = self.jss.get(url)
  File "/Library/Python/2.7/site-packages/python_jss-0.4.2-py2.7.egg/jss/jss.py", line 196, in get
    response = self.session.get(url)
  File "/Library/Python/2.7/site-packages/python_jss-0.4.2-py2.7.egg/jss/contrib/requests/sessions.py", line 469, in get
    return self.request('GET', url, **kwargs)
  File "/Library/Python/2.7/site-packages/python_jss-0.4.2-py2.7.egg/jss/contrib/requests/sessions.py", line 443, in request
    prep = self.prepare_request(req)
  File "/Library/Python/2.7/site-packages/python_jss-0.4.2-py2.7.egg/jss/contrib/requests/sessions.py", line 374, in prepare_request
    hooks=merge_hooks(request.hooks, self.hooks),
  File "/Library/Python/2.7/site-packages/python_jss-0.4.2-py2.7.egg/jss/contrib/requests/models.py", line 304, in prepare
    self.prepare_url(url, params)
  File "/Library/Python/2.7/site-packages/python_jss-0.4.2-py2.7.egg/jss/contrib/requests/models.py", line 358, in prepare_url
    "Perhaps you meant http://{0}?".format(url))
jss.contrib.requests.exceptions.MissingSchema: Invalid URL u'None/JSSResource/distributionpoints': No schema supplied. Perhaps you meant http://None/JSSResource/distributionpoints?
>>> 

[acodega]

I typed the URL manually, now I'm going to type the username manually and then the password and see if I can get it to not fail like I thought I did earlier.

[eahrold]

Sorry I was a little unclear, what i meant is to do all the export commands then enter the python interpreter, then....

>>> print os.environ.get("JSS_URL")

then copy the results of that and paste it into this...

>>> j = jss.JSS(url="paste_the_valuehere", user=os.environ.get("JSS_USERNAME"), password=os.environ.get("JSS_PASSWORD"))

Since you said you had it working here https://github.com/lindegroup/autopkgr/issues/221#issuecomment-65237668 this should also work.

[acodega]

Right that's what I did, and it failed in a slightly different way. (Yay progress!)

j = jss.JSS(url=os.environ.get("https://jss.jamfcloud.com/[instance]/"), user=os.environ.get("JSS_USERNAME"), password=os.environ.get("JSS_PASSWORD"))

[eahrold]

Hey Adam, Actually it's slightly different. You did this

>>> j = jss.JSS(url=os.environ.get("https://jss.jamfcloud.com/[instance]/"), user=os.environ.get("JSS_USERNAME"), password=os.environ.get("JSS_PASSWORD"))

but what I mean is this

>>> print url=os.environ.get("JSS_URL")
https://jss.jamfcloud.com/instance

## then copy the above printed value and paste it directly as the url= value (no os.envrion.get() business)
>>> j = jss.JSS(url="https://jss.jamfcloud.com/instance", user=os.environ.get("JSS_USERNAME"), password=os.environ.get("JSS_PASSWORD"))

The first...

url=os.environ.get("https://jss.jamfcloud.com/[instance]/")

Would set the url to the environmental value for the KEY "https://jss.jamfcloud.com/[instance]/", but there is no environment value for the KEY.

See the difference?

[acodega]

Ahh okay, it may take some time but I'll let you know when I get results.

[eahrold]

Oops, Sure is atypo, here's the correct thing...

print os.environ.get("JSS_URL")

[acodega]

I had deleted my comment since I was able to get a co-worker to help find the issue.

We discovered the problem is if your JSS URL has a / at the end, then what ends up happening is the URL turns into https://jss.jamfcloud.com/[instance]//[relativeURL], the double slashes causes the 404.

[eahrold]

WOW! That's good to know. Normally that shouldn't matter, but quite often theory and practice diverge, in fact I actually brought that up with Shea, I'm sure he'll want to know that this will in fact cause 404 so it can get addressed.

So is AutoPkg(r) working now with the trailing slash removed?

[acodega]

Well now I'm having problems mounting the DP

I tried this on my DP and also on my local system

With SMB:

12/8/14 2:27:31.258 PM AutoPkgr[57613]: [DEBUG] Error [1] Error running recipes 
 mount_smbfs: server connection failed: Connection refused
A Python exception occurred during the execution of autopkg, see the console log for more details.

[ERROR] subprocess.CalledProcessError: Command '['mount', '-t', 'smbfs', '//scrubbed:scrubbedpwd@192.ip.1.1:139/CasperShare', '/Volumes/ScrubbedITCasperShare']' returned non-zero exit status 68

Same with AFP:

12/8/14 3:34:06.780 PM AutoPkgr[58574]: [DEBUG] Error [1] Error running recipes 
 mount_afp: AFPMountURL returned error 2, errno is 2
A Python exception occurred during the execution of autopkg, see the console log for more details.

[ERROR] subprocess.CalledProcessError: Command '['mount', '-t', 'afp', 'afp://scrubbed:scrubbedpwd@192.ip.1.1:139/CasperShare', '/Volumes/ScrubbedITCasperShare']' returned non-zero exit status 2

I can mount the drive manually without errors, mounting it myself and then running a recipe check still doesn't work.

[eahrold]

Looks like a port issue.

SMB could be using 139 which is more widely used in a windows environment, but if your mounting it from Finder it's likely using port 445. I've seen strange behavior accessing a Mac Server via SMB from a Mac client with 139, and 445 works much better.

As far as AFP, that shouldn't be using 139 at all, so that explains why that failed (but it's strange how it got to be that way).

Did you enter these manually or were they retrieved from the JSS? If manually try setting the port explicitly.

[acodega]

I think there's a problem with the DP share, which doesn't make a lot of sense but Casper Admin can't mount the share automatically either. Will investigate and report.

Is the trailing slash issue a part of AutoPkg or how AutoPkgr hands the variable?

[eahrold]

The trailing slash is ultimately part of python-jss. I'll re-open my issue there. We can easily enough fix it in AutoPkgr, and probably will just to be on the safe side in 1.2.

Out of curiosity, do you see the same behavior when you enter the double slashed url into a web browser?

Is the DP suppose to be an AFP share or a SMB share? We've seen situations where once upon a time the share was set up in the JSS as SMB (for example), then changed to AFP however the Port was still explicitly set to 139 for that DP.

[acodega]

That's what I'm leading towards too, thanks for pre-confirming for me that switching a share type can cause issues. Going to remake the share and recheck.

[eahrold]

Hey @acodega, Get to the bottom of this?

[acodega]

I was just working on it again this morning when you pinged me.

I updated AutoPkgr and JSSImporter and it ran successfully once. It created the policy and mounted and unmounted the DP successfully. That was with the DP credentials manually entered.

I tried using the auto populated credentials and I get this error now, I tried going back to manually entered credentials and get the same error:

12/19/14 11:40:16.754 AM AutoPkgr[25730]: (2/2) Processing Firefox.jss...
12/19/14 11:40:30.979 AM AutoPkgr[25730]: (FULL AUTOPKG TRACEBACK) 
  File "/usr/local/bin/autopkg", line 1469, in <module>
    sys.exit(main(sys.argv))
  File "/usr/local/bin/autopkg", line 1463, in main
    exit(subcommands[verb]['function'](argv))
  File "/usr/local/bin/autopkg", line 1287, in run_recipes
    autopackager.process(recipe)
  File "/Library/AutoPkg/autopkglib/__init__.py", line 466, in process
    self.env = processor.process()
  File "/Library/AutoPkg/autopkglib/__init__.py", line 295, in process
    self.main()
  File "/Library/AutoPkg/autopkglib/JSSImporter.py", line 594, in main
    suppress_warnings=suppress_warnings)
  File "/Library/Python/2.7/site-packages/python_jss-0.5.3-py2.7.egg/jss/jss.py", line 141, in __init__
    self.distribution_points = distribution_points.DistributionPoints(self)
  File "/Library/Python/2.7/site-packages/python_jss-0.5.3-py2.7.egg/jss/distribution_points.py", line 89, in __init__
    self.dp_info = self.jss.DistributionPoint().retrieve_all()
  File "/Library/Python/2.7/site-packages/python_jss-0.5.3-py2.7.egg/jss/jss.py", line 295, in DistributionPoint
    return self.factory.get_object(DistributionPoint, data)
  File "/Library/Python/2.7/site-packages/python_jss-0.5.3-py2.7.egg/jss/jss.py", line 441, in get_object
    result = self.jss.get(url)
  File "/Library/Python/2.7/site-packages/python_jss-0.5.3-py2.7.egg/jss/jss.py", line 174, in get
    self._error_handler(JSSGetError, response)
  File "/Library/Python/2.7/site-packages/python_jss-0.5.3-py2.7.egg/jss/jss.py", line 161, in _error_handler
    raise exception
jss.exceptions.JSSGetError: JSS Error. Response Code: 404   Response" 
12/19/14 11:40:30.979 AM AutoPkgr[25730]: [DEBUG] Error [1] Error running recipes 
 A Python exception occurred during the execution of autopkg, see the console log for more details.

[ERROR] jss.exceptions.JSSGetError: JSS Error. Response Code: 404   Response" 
12/19/14 11:40:31.005 AM AutoPkgr[25730]: [DEBUG] Nothing new was downloaded.

I am going to try manually running the Python commands like we did before.

[eahrold]

Now you're back to a JSS issue, not a DP issue. Are you sure the JSS url does not have a trailing slash, as you discovered breaks it.

Did you remove the auto-populated DP's?

[acodega]

The auto populated DPs are okay, right now it's only a URL issue. I can't enter it by removing the slash on the Folders & Integrations tab but I can change it by running

defaults write com.github.autopkg JSS_URL https://jss.jamfcloud.com/[instance]

From the command line.

[eahrold]

Sorry, what is it you can't enter by removing the slash on the F&I tab. You mean that when you enter it there it doesn't update the defaults?

[acodega]

Hmm, I wish I had more consistent behavior.

Right now if I enter my URL without a trailing slash, it writes the JSS_URL value without a trailing slash but in the GUI it shows my url WITH a trailing slash and also reports in the debug log that the API credentials may not be correct.

However, AutoPkgr is running perfectly for me right now. Checking for updates, downloading them to the DP and creating the policy on the JSS.

I believe this issue was originally related to trailing slashes, I'll mark it as solved for now but not sure what the permanent fix should be, if it's in AutoPkgr or another component.

[eahrold]

So I'm guessing there are two things at play here. Right now AutoPkgr won't update the 'defaults' value for the JSS_URL, API_USER, and API_PASSWORD until the connect button is pressed and the connection is verified. So simply updating the text field won't automatically update the defaults key. We set it up like this to ensure the credentials checked out prior to setting, but in the upcoming 1.2 release we're going to be more assertive when setting these values and set them when entered.

As far as why the text field is not representing the value set in the defaults, the only thing I can think there is that AutoPkgr hasn't been relaunched since setting that (using terminal), because at launch is the only time that would be picked up if set externally.

In 1.2 a lot of this behavior is going to get changed so I think you'll find it more consistant.

Like alway thanks for the report, and for keeping us on our toes.