pyracf

PyRACF: A Python module for analyzing RACF security

PyRACF is a powerful Python module that simplifies the parsing and querying of any RACF database, providing an efficient and intuitive way to analyze security setups on IBM Z systems. By consuming the IRRDBU00 unload, PyRACF generates "Panda DataFrames" for each 'recordtype', which allow for seamless manipulation and analysis of the data.

Pandas is a powerful data manipulation library in Python that allows for easy querying of the DataFrames generated by PyRACF. With Pandas, you can perform complex queries on the security data to extract meaningful insights into the security posture of your system.

PyRACF's support for saving and loading pickle files makes it easier than ever to work with large RACF datasets, giving you the ability to perform comprehensive analyses of your security setup.

For more information on the various records, please refer to the IBM documentation on IRRDBU00 record types and the record formats produced by the database unload utility. The DataFrames generated by PyRACF feature the same 'fieldnames' as outlined in the documentation, ensuring consistency and accuracy in your analyses.

To get started with PyRACF, install it using pip install pyracf or explore the source code on GitHub. Use PyRACF to take control of your security data and protect your IBM Z systems from threats.

Updates

0.8.7 (fixes for pickles, pytest, wiki)

• grouptree and ownertree are now properties, no longer callables
• accept * as a literal value in gfilter( )
• r.connect('SYS1') and r.connect(None,'IBMUSER') return one level index
• less contentious column name ALL_USER_ACCESS replaces EFFECTIVE_UACC
• speed up single profile methods
• Single value selections return dataframe with columns again
• giveMeProfiles, generic2regex are now 'internal' (_) functions

0.8.5 (fixes for pickles, pytest, wiki)

• parse_fancycli now creates empty data frames for pickles it could not find
• index added to data frames from old pickles, not for pickles that already have index fields
• pytest framework for QA in development cycle, initial tests ensure attributes are the same with all 3 methods to obtain RACF profiles
• wiki https://github.com/wizardofzos/pyracf/wiki

0.8.3 (tree print format for grouptree and ownertree)

• msys.grouptree and msys.ownertree are now properties instead of callables print as unix tree or simple format, e.g. print(msys.ownertree) default format looks like unix tree, change with msys.ownertree.setformat('simple') dict structure accessible through .tree attribute
• .connect('group') and .connect(None,'user') return a (single level) Index with user IDs, resp., groups, connected to the given entity this helps with queries that test group membership
• add IDSTAR_ACCESS and ALL_USER_ACCESS to .datasets and .generals with, resp., permitted access on ID(*) and the higher value of UACC and IDSTAR_ACCESS.
• fixed: correlate also builds internal tables from saved pickles

0.8.2 (property for most application segments)

• application segments for dataset, group and user entities are avaible with entity prefix, e.g., msys.userTSO, msys.datasetDFP, msys.groupOMVS
• system related application segments from general resources are published without prefix, e.g., msys.STDATA or msys.MFA
• old properties msys.installdata and msys.userDistributedMappings are replaced by userUSRDATA and userDistributedIdMappings
• most of these properties are automatically generated

0.8.0 (acl, gfilter, rfilter methods)

• selection method gfilter (supports RACF generic patterns) and rfilter (for regex patterns) supports as many parameters as there are index columns in the frame
• reporting method acl, produces frame with access list, may be used on the entity frames or on the access frames
• internal frames _connectByGroup and _connectByUser, as alternate index on connectData
• internal frames _grouptreeLines and _ownertreeLines that return all groups up until SYS1 (or upto a user ID)
• correlate() invoked automatically by parse() and fancycli()

0.7.1 (General instead of Generic)

• fixed: Generic should be General resource profiles, variables and methods renamed
• Mapping between IRRDBU00 record types and variables centralized in a dict
• global constants related to record types removed
• parsed records internally stored by (decimal) record type, instead of by name
• add method to retrieve parsed record count for given name
• _offsets now a RACF class attribute
• fixed: pickles with a prefix were selected when loading pickles with no prefix
• fixed: status property crashed when used before parse() method used, math.floor call is now conditional
• fixed: record type '0260' in offset.json was malformed
• updated offsets.json from https://www.ibm.com/docs/en/SSLTBW_3.1.0/com.ibm.zos.v3r1.icha300/format.htm etc
• getOffsets.py to update the json model
• fixed: RACF documentation contains incorrect record type 05k0
• all known record types parsed and loaded into DataFrames
• index columns assigned to all DataFrames, assigned by new correlate() method
• new method correlate() to increase speed of subsequent data access, used after parse() or loading of pickles
• new selection methods similar to user() and group(), that work on index fields. when a parameter is given as None or '**', elements matching the other parameters are returned: datasetPermit and datasetConditionalPermit, with parameters profile(), id() and access() generalPermit and generalConditionalPermit, with parameters resclass(), profile(), id() and access() connect with parameters group() and user()
• added GPMEM_AUTH to connectData frame, consolidating all connect info into one line

0.6.4 (Add 0209)

• Added 0209 recordtype to parser. (userDistributedMapping)

0.6.3 (Add fields)

• Added missing USBD_LEG_PWDHIST_CT, USBD_XPW_PWDHIST_CT, USBD_PHR_ALG, USBD_LEG_PHRHIST_CT, USBD_XPW_PHRHIST_CT, USBD_ROAUDIT and USBD_MFA_FALLBACK to Users dataframe

0.6.2 (Fix XLSX Creation)

• With newer versions of XlsxWriter there's no more .save(). Changed to .close()
• Pinned pandas and XlsxWriter versions in setup.py

0.6.1 (Bug free?)

• XLS generation fully functional again (also for z/VM unloads)
• Oprhan detection working again
• Conditional Dataset Access Records now parsing correctly
• Conditional Dataset Access now correctly pickled :)
• Fixed parsing of GRCACC records (had misparsed AUTH_ID)
• Conditional Generic (General) Records now with correct column name (GRCACC_CLASS_NAME)

0.5.4 (Even more recordtypes!!)

• new property: genericConditionalAccess. Will show GRCACC records.
• Fixed some nasty 'default recordtypes' bugs

0.5.0 (Pickle FTW!)

• new function: save_pickles(path=path, prefix=prefix). Will save all parsed dataframes as pickles (/path/prefixRECORDTYPE.pickle)
• Can now initialize RACF object from pickle-folder/prefix. To reuse earlier saves pickle files. See examples below
• parse_fancycli now has two optional arguments (save_pickles and prefix) to also save pickle files after parsing to the directory as specified in save_pickles. The prefix argument is only useed with save_pickles isn't False

0.4.5 (Fix Community Update Bug, thanks @Martydog)

• Group Connections now actually usable :)

  0.4.4

• Internal constants for all recordtypes

• Improved 'parse_fancycli()'

0.4.3 (Community Update, thanks @Martydog)

• Add User Group Connections record 203
• Add User Installation Data record 204

0.4.2

• Now XLS generation has more checks (fails gracefully if not all required records parsed, works when only genericAccess parsed)
• Same for Orphan detection
• Recordtype 0503 (General Resource Members/genericMembers) added

Parsing IRRDBU00 unloads like a boss

>>> from pyracf import RACF
>>> mysys = RACF('/path/to/irrdbu00')
>>> mysys.parse()
>>> mysys.status
{'status': 'Still parsing your unload', 'lines-read': 200392, 'lines-parsed': 197269, 'lines-per-second': 63934, 'parse-time': 'n.a.'}
>>> mysys.status
{'status': 'Ready', 'lines-read': 7137540, 'lines-parsed': 2248149, 'lines-per-second': 145048, 'parse-time': 49.207921}

Using Pickle Files

>>> from pyracf import RACF
>>> mysys = RACF('/path/to/irrdbu00')
>>> mysys.parse_fancycli(save_pickles='/tmp/pickles', prefix='mysys-')
>>> hash(mysys.groups.values.tobytes())
-8566685915584060910

Then later, you don't need to parse the same unload again, just do:

>>> from pyracf import RACF
>>> mysys = RACF(pickles='/tmp/pickles', prefix='mysys-')
>>> hash(mysys.groups.values.tobytes())
-8566685915584060910

All functions

Example use-case

Get all users that have not logged in (on?) since January 1st 2022. And print userID and last logon...

import time
from pyracf import IRRDBU

mysys = IRRDBU('/path/to/irrdbu00')
mysys.parse()
while mysys.status['status'] != 'Ready':
    time.sleep(5)
selection = mysys.users.loc[mysys.users.USBD_LASTJOB_DATE<="2022-01-01"][['USBD_NAME','USBD_LASTJOB_DATE']]
for user in selection.values:
  print(f"Userid {user[0]}, last active: {user[1]}")

Create a neat XLSX

import time
from pyracf import IRRDBU
mysys = IRRDBU('/path/to/irrdbu00')
mysys.parse()
while mysys.status['status'] != 'Ready':
    time.sleep(5)
mysys.xls('/path/to/my.xlsx')

Print z/OS UNIX profiles

mysys.general('FACILITY', 'BPX.SUPERUSER')
mysys.general('FACILITY', 'BPX.**')  # show only the FACILITY BPX.** profile
mysys.general('FACILITY')  # show all FACILITY profiles

mysys.generals.gfilter('FAC*', 'BPX.**')  # show all BPX profiles
mysys.generals.gfilter('UNIXPRIV') # print all in UNIXPRIV

Show group information

mysys.connect('SYS1')            # users connected to SYS1 groups
mysys.connect('**','IBMUSER')    # groups IBMUSER is connected to

mysys.connectData.gfilter('SYS*','IBM*') # connects using patterns
mysys.connectData.rfilter('SYS[AB].*','PROD.*') # regex with alternatives
mysys.connectData.query("USCON_GRP_SPECIAL=='YES' & USCON_REVOKE=='YES'")  # group special revoked
mysys.connectData.query("GPMEM_AUTH==['CONNECT','JOIN']")  # users with connect authorities

mysys.users.query("index in @mysys.connect('SYS1').index")  # user details of users connected to SYS1

Show access list information

mysys.datasetPermit('SYS1.**')    # IDs permitted on SYS1.**
mysys.datasetPermit(id='IBMUSER', access='ALTER')    # where is IBMUSER permitted
mysys.datasetPermit(id='*')       # where is ID(*) permitted

mysys.datasets.gfilter('SYS1.**').acl(access='ALTER')    # IDs ALTER access on any SYS1 dataset profile
mysys.datasets.gfilter('SYS%.**').acl(allows='UPDATE')    # IDs with access that allows UPDATE
mysys.datasets.gfilter('SYS%.**').acl(allows='UPDATE', resolve=True)    # groups and user permits combined 
mysys.datasets.gfilter('PROD.**').acl(permits=False, admin=True)    # who can change groups or profiles to change access on PROD data sets
mysys.generals.gfilter('XFAC*', 'CKR.**').acl() # permits on zSecure Admin/Audit profile

mysys.datasets.query("ALL_USER_ACCESS=='UPDATE'")    # UACC or ID(*) set to UPDATE

Show group tree information

print(msys.grouptree)  # group - superior group - subgroups in UNIX tree format
print(msys.ownertree.setformat('simple'))  # group - OWNER in simple format
msys.grouptree.tree  # get the dict

Updates

In this version we introduced IRRRDBU as an alternative to RACF. Examples have been updated. The RACF class from previous version is still available, but you're advised to change this to IRRDBU, as future version will have another user of the RACF class.

Contribute to PyRACF

If you've some additions and/or bugfixes, feel free to fork this repository, make your additions and fire a pull request.