search

EliziumNet / RexFs

Regular expression based Bulk File/Directory renamer
MIT License
4 stars 2 forks source link
bulk bulk-operation bulk-rename powershell powershell-module regex rename-files renamer

readme

:shield: Elizium.RexFs

_Regular expressions based file system operations module with Bulk File/Directory renamer powered by :nazar_amulet: Elizium.Loopz and colouring provided by [:rainbow: Elizium.Krayola__](https://github.com/EliziumNet/Krayola)___

A B A B A B A B Join the chat at https://gitter.im/EliziumNet/rexfs

Table Of Contents

Introduction

The module can be installed using the standard install-module command:

PS> install-module -Name Elizium.RexFs

The following dependencies will be installed automatically

:high_brightness: Rename-Many (remy) is a flexible regular expression based bulk rename command. In order to get the best out of this command, the user should become familiar and skilled in writing regular expressions.

:pushpin: Note, that it is recommended that this page is read in sequential order, in particular because the first usage scenario to be explained: Move Match, includes more screen snapshots and examples to help explain the concepts, which is not repeated to the same detail in subsequent usage scenarios. See also the Parameter Reference, for a more detailed explanation of each parameter and Safety Features to see how to Unlock the command so it becomes effective.

There are multiple modes of operation that Rename-Many runs in, which are Update/Move/Cut/Appendage. On top of this, a developer can extend its capabilities by providing a custom Transform and/or build Higher Order Commands.

Mode DESCRIPTION
Move Match Move a regular expression match to an anchor
Update Match Replace a regular expression match
Cut Match Remove a regular expression match
Add Appendage Add a fixed token to Start or End of item
Transform Apply a custom transform to perform rename

:rocket: Quick Start

Rename-Many works by receiving it's input from the pipeline and renaming these file system items according to parameters specified. The benefit of using this command comes when there is a need to rename multiple items according to the same criteria. This criteria amounts to specifying regular expressions. The 2 most commonly used scenarios are:

Quick Move

Moves a regex match to another point, known as an anchor.

:one: Let's say we have a directory containing a collection of log files, some of which are not named according to a fixed form eg they contain a date which is not at the end of the file name. We can bulk rename the rogue files with a command like:

gci ./*.log | Rename-Many -Pattern '(?\\d{2})-(?\\d{2})-(?\\d{4})' -End -WhatIf

Note the following:

So an directory named:

'27-03-1997 Orbital - The Middle Of Nowhere Tour Pics'

would be renamed to

'Orbital - The Middle Of Nowhere Tour Pics27-03-1997'

:two: Extending this example a little further, let's say we also want to reformat the date so it's in ISO format (yyyy-mm-dd). We can achieve this by specifying the formatter parameter -With (With is used for move operations, whereas Paste is used for update operations) as illustrated below:

... | remy -Pattern '(?\\d{2})-(?\\d{2})-(?\\d{4})' -End -With '\${y}-\${m}-\${d}' -WhatIf

So that same directory referenced in example :one: would now be renamed to:

'Orbital - The Middle Of Nowhere Tour Pics 1997-03-27'

You should note that now we are using a formatter, we can repair the fact that the new date placement now has a space separator after 'Pics' (the previous end of the item's name) as opposed to the match being inserted verbatim as in example :one:.

In practice, there are a few more subtleties that need to be accounted for when performing bulk renames and the reader will discover this when reading the rest of this documentation. One such nuance is that moving the date like this may require the insertion of additional content to get the graceful result we require, eg wrap the date in brackets, or insert a dash before the date, as shown here, ... etc.

Quick Update

Replace a regex match in it's present location with alternative content.

Continuing with our log files theme, let's say we just want to adjust the date format. So any date that appears as 'dd-mm-yyyy' we would like to adjust it to ISO format 'yyyy-mm-dd' instead. We can do this using the update mode of Rename-Many:

... | remy -Pattern '(?\\d{2})-(?\\d{2})-(?\\d{4})' -Paste '\${y}-\${m}-\${d}'

Hopefully, the above examples have given a quick insight in the operation of the command, enough to get started with. Also note, to begin with, you can initially stick to using static patterns instead of more complex regex sequences for the more simple rename operations, eg rename the presence of the word 'CCY' to 'Currency', by using a pattern of 'CCY' and -With/-Paste of 'Currency'.

:warning: An important point of note that the user must take heed of early on is, all formatter parameters (-Drop, -Paste and -With), MUST be specified with single quotes not double quotes (see Formatter parameters must use single quotes for more details).

:sparkles: General Concepts

:gem: Safety features

Rename-Many is a powerful command and should be used with caution. Because of the potential for accidental misuse, a number of protections have been put in place:

:gem: Occurrence

All regular expression parameters as listed below ...

Regex Parameter Alias DESCRIPTION
Anchor a Move a match to an anchor
AnchorEnd ae Move a match to an anchor or to end if anchor fails match
AnchorStart as Move a match to an anchor or to start if anchor fails match
Copy co Make a copy of this match for formatter reference
Cut :heavy_multiplication_x: Remove this match without a replacement
Except x Filter out items that match
Include i Filter in items that match
Pattern w Replace or Move this match

... are all declared as arrays. This allows the user to augment the regular expression with an additional value denoting which match occurrence is in effect. This value can be either numeric which denotes which match to select or 'f' for the first match or 'l' indicating the last match. The exception to the use of the Occurrence value is with -Include/-Except, which are filtering parameters (see Filtering). Since they are used for filtering, the Occurrence value is irrelevant, so should not be supplied.

So for example -Pattern can be specified as:

Rename-Many -Pattern 'foo', 2 ...

This means that the second occurrence of the match 'foo' should be taken as the token match for each item being renamed.

:gem: Escaping

If a regex parameter needs to use a regular expression character as a literal, it must be escaped. There are multiple ways of doing this:

The above 2 approaches escape the entire string. The second approach is more concise and avoids the necessary use of extra brackets and $.

This third method is required when the whole pattern should not be subjected to escaping.

:gem: Filtering

Generally, the user must indicate which items are to be renamed using the pipeline. Any command can be used to select file system items (directories or files), but typically Get-ChildItem would be used and the result piped to Rename-Many. Get-ChildItem contains a -Filter parameter but this can only filter as a blob using wildcards where appropriate, but can not filter by a regular expression. The user could use an extra pipeline stage using Where-Object eg:

Get-ChildItem -LiteralPath ./ -File -Filter '*.log' | Where-Object { $_.Name -match 'bar' } | Rename-Many ...

However, this command is starting to get quite long (even if we used the ? alias for Where-Object). So instead, Rename-Many contains regex filters via the -Include/-Except parameters:

Get-ChildItem -LiteralPath ./ -File -Filter '*.log' | Rename-Many -Include 'bar' ...

and similarly for -Except.

Any items filtered out 'inband' (by the filter parameters on Rename-Many as opposed to 'out of band' filtering applied to the previous stage of the pipeline, eg the filter parameter on Get-ChildItem), will be counted as a 'Skipped' item in the Summary displayed at the end of the rename batch.

:gem: Formatter Parameters

The following parameters are known as formatters. This means that they are strings which contain the replacement text for the match. The formatter can also reference named (or numbered) group references defined in the non-filtering regex parameters:

Formatter Parameter Alias DESCRIPTION
With w used when performing Move
Drop dr for Move operations where the Pattern match is replaced by drop content
Paste ps used when performing in place Update

So given the following as an example (not all parameters have been defined so do not take this as a literal example)

... | Rename-Many -Pattern '(?\<y>\d{4})-(?\<m>\d{2})-(?\<d>\d{2})' -With '\${d}-\${m}-${y}'

we can see that inside the -With formatter, there are references to named group captures ('d', 'm', 'y') that are defined inside the -Pattern regex.

:gem: General Parameters

The following parameters belong to all Rename-Many parameter sets:

(The parameters marked as Interactive are those which are meant to be used interactively. The other parameters are intended for use by developers needing to expand the renaming possibilities of the command.)

General Parameter Alias Interactive (non Developer) DESCRIPTION
Condition :heavy_multiplication_x: :x: A custom predicate script-block that filters pipeline items
Context :heavy_multiplication_x: :x: A PSCustomObject with options to customise Rename-Many
Directory d :heavy_check_mark: Filter for Directory items only
Diagnose dg :heavy_check_mark: WhatIf mode with added diagnostic info
End e :heavy_check_mark: Move Pattern
Except x :heavy_check_mark: Filter out items that match this regex
File f :heavy_check_mark: Filter for File items only
Include i :heavy_check_mark: Filter out items that do not match this regex
Start s :heavy_check_mark: Move matched Pattern to start of items name
Top t :heavy_check_mark: Process the first n items only
Transform :heavy_multiplication_x: :x: A script-block to perform custom rename operation
Whole :heavy_multiplication_x: :heavy_check_mark: Applies 'whole word' match to regex parameters

:gem: Post Processing

During the renaming process, an item maybe renamed resulting in unfavourable characteristics in the resultant file/directory name. Eg, it is not wise to leave a file name with trailing or leading spaces. The following are the characteristics that are automatically fixed by a post processing operation:

For Trim and Spaces, the post processing simply involves removal of the unwanted spaces. For MissingCapture this process involves the removal of any string of the form:

${some-variable}

which occurs as a result of a named group reference defined in a formatter parameter (-With/-Paste) not being populated due to application of a regex parameter to the item and it not matching. An example of this occurring is with the use of a Hybrid Anchor (-AnchorStart/-AnchorEnd), where the user specifies a pattern that includes a named group capture, but for some items, this match may fail (so by design, the match would fall back to being moved to start or end), but the formatter parameter -With contains a reference to that named capture group. When this match fails, the reference '${some-variable}' still remains, which is removed by the post processing operation.

The Dashes rule ensures that we never allow a poor dash formation to occur, something like '- -' which can typically occur if a pattern is designed to work on item names that are denoted by fields split by dashes and an operation moves a field from 1 location to another leaving behind unsightly dash/space sequences. These are normalised away by replacing with a tidy single ' - '.

These post-processing operations are made explicit in the UI because sometimes they occur as a result of a mal-formed regular expression that can be automatically fixed, but doing so may lure the user into thinking they're specifying the regex pattern correctly when in fact a minor correctable mistake has been made. The command aims to be explicit about such issues rather than fix them silently.

The following shows an example of the MissingCapture operation being applied and how it shows up in the output:

picture

:gem: Signals

As the saying goes, 'a picture is worth a thousand words'. This is particularly cogent when eye-balling a stream of repetitive content generated as a result of an iterative operation such as the result of Rename-Many. Viewing and trying to process a wall of mono-coloured un-structured content can be difficult. This was the rationale behind the use of emojis (generalised into the concept of a Signal in commands, see Loopz) and of another Elizium PowerShell module Krayola. The combination of coloured, consistently structured text with the use of emoji based signals is intended to aid human readability of repetitive content.

:gem: The Replacement Process

For those parameter sets that require the -Pattern parameter (which is most of them; NoReplacement which uses the -Cut parameter, does not), the content that is matched by the Pattern, is removed prior to applying other regex parameters.

It is useful for the user to think about this when composing regex patterns. So for example, given the following input source:

01-From-My-Mind-To-Yours

... a -Pattern defined as:

'\d{2}-'

results in first, the removal of the -Pattern match leaving this behind:

From-My-Mind-To-Yours

It is to this remainder that all other regex parameters are applied.

:gem: Saved Undo Scripts

The undo facility enables a rename batch to be reversed. The location of the scripts is displayed in the rename summary. By default, scripts are saved to '.elizium' under the home ($Home) directory, but this can be overridden.

To change this path, the user should define either an absolute or relative path in the environment variable 'ELIZIUM_PATH'. Relative paths are relative to the $Home directory.

:sparkles: Move Match

Move Parameter Alias DESCRIPTION
Anchor a Move a match to an anchor
AnchorEnd ae Move a match to an anchor or to end if anchor fails match
AnchorStart as Move a match to an anchor or to start if anchor fails match
End e Move match to end
Start s Move match to start
With w Formatter used when performing Move

Moves a match from it's current location in an item to a target location known as the Anchor. The anchor itself is a regular expression. All of the parameters in this section, with the exception of -With are mutually exclusive (to see confirmation of this, the user can use the Parameter Set Tools in particular the command Show-ParameterSetInfo (ships), which reveals that they are indeed the unique parameters in their respective parameter sets).

In the following walk-throughs, example invocations are preceded with a :heavy_minus_sign: to indicate a solution that has some scope for improvement. Subsequent to this will be further discussion on how to improve the command and those which are deemed satisfactory are marked with :heavy_plus_sign:. Sometimes, a command does not work in the desired way. These examples are highlighted by a :x:.

:gem: Move to Anchor

Move a regex match identified by -Pattern from the items' name from its current location to a location specified by the -Anchor regex pattern.

Let's say we have a bunch of audio files which are currently named in the form:

\'<DISC-NO>-\<TRACK-NO>_\<TRACK-NAME>.mp3'

(the underscore used in all examples are meant to represent a single space)

... and we wish to move the DISC-NO to after the TRACK-NO. In this case, the DISC-NO would be the subject of the -Pattern match and the TRACK-NO would be the -Anchor.

So an initial attempt of the command could be:

:heavy_minus_sign: Rename-Many -Pattern '\d{2}' -Anchor '\d{2}' -WhatIf

gci ... | Rename-Many -Pattern '\d{2}' -Anchor '\d{2}' -WhatIf

:pushpin: At the end of this section a final version of the command will be illustrated, but we will get there in small steps, so that little nuances can be explained fully.

Focusing on a single file item in the batch being: '02-09 Radio Stars.mp3':

:heavy_minus_sign: Rename-Many -Pattern '\d{2}' -Anchor '\d{2}' -WhatIf

-0902 Radio Stars.mp3

Although the order of the DISC-NO and TRACK-NO have been swapped around, this is almost certainly not what we would want. We need to maintain the dash in between them. We can't include the dash inside the -Pattern because that would just result in '0902-'. This is where the -With formatter parameter and -Relation comes into play. We can format the replacement text:

:heavy_minus_sign: Rename-Many -Pattern '\d{2}' -Anchor '\d{2}' -With '\${_a}-$0' -WhatIf

-09-02 Radio Stars.mp3

This is starting to get better, but there is still a problem. We now have a stray leading dash, but before discussing that issue, the contents of the -With parameter needs explaining. Formatter parameters can access whole regex captures defined by other parameters and/or named/numeric capture groups defined within them. So in this example '\$0' represents the whole -Pattern match which evaluates to '02' and ${_a} represents the whole -Anchor match which evaluates to '09'.

So back to the issue at hand, being the leading stray '-'. We could solve this 1 of 2 ways

:radio_button: -Pattern '(?\<disc>\d{2})-' -Anchor '\d{2}' -With '\${_a}-${disc}' -WhatIf

09-02 Radio Stars.mp3

:radio_button: -Pattern '\d{2}' -Anchor '-(?\<track>\d{2})' -With '${track}-\$0' -WhatIf

09-02 Radio Stars.mp3

Finally, we might decide that the \<TRACK-NO>-\<DISC-NO> sequence needs to be more clearly separated from the \<TRACK-NAME>, so an extra ' - ' is inserted into -With, let's say using technique :one: above (but equally applies to :two:):

:heavy_plus_sign: Rename-Many -Pattern '(?\\d{2})-' -Anchor '\d{2}' -With '\${_a}-${disc} - ' -WhatIf

09-02 - Radio Stars.mp3

Now that we have our somewhat finalised version, lets see how this looks in a batch:

(actually, the screen shot below uses the Top parameter to reduce the number of items processed, for brevity)

picture

It can be seen that for each item renamed, the new name is displayed in red, with the original name displayed in white. Next to the new name, supplementary info is displayed, including 'Post (Spaces)' (we'll come to this a little later) and a -WhatIf indicator.

At the beginning of the batch, a title is shown with the Locked status highlighted. At the end of the batch, the rename summary is shown, displaying the value of key parameter values and some stats.

The 'Post (Spaces)' previously mentioned, indicates we have made a slight formatting error that has been fixed automatically for us by Post Processing. In this case, we have created a -With formatter that results in consecutive spaces in the resultant rename.

So looking at our definition of -With again:

-With '\${_a}-${disc} - '

That space at the end is our issue. There is already a space preceding the \<TRACK-NAME> which was not captured by either -Pattern or -Anchor, so we don't need to insert another. So adjusting this to be

-With '\${_a}-${disc} -'

... without the trailing space, fixes the problem:

:heavy_plus_sign: Rename-Many -Pattern '(?\<disc>\d{2})-' -Anchor '\d{2}' -With '\${_a}-${disc} -' -Top 10 -WhatIf

picture

The Post Processing is there to watch our back by automatically enforcing desirable rules and makes the command less pedantic in its operation.

In other more complicated rename batches, we might (and probably will) encounter a scenario where the named group captures defined are not doing what we expected. In this case, we can run the command with diagnostics enabled via the -Diagnose parameter.

For this example, when -Diagnose is specified, we can see the value of named capture groups:

picture

Focusing on the first entry, item named '01-01 Autobahn.mp3', we can see the diagnostics entry:

"[🧪] Pattern" => "([☂️] \<disc>)='01', ([☂️] <0>)='01-'"

This tells us that -Pattern contains named group reference(s), in this case:

This is quite a convenient and tidy example, because all the input items are of identical form, but this is not always the case. Let's assume, the first entry in this list: '01-01 Autobahn.mp3' is not named that way, instead it is '01-0 Autobahn.mp3' so the track number is now just a single digit '0'.

In this case, the -Pattern will match because there is still a 2 digit sequence, but the -Anchor will no longer match. This results in this item not being renamed and this is indicated in the output:

picture

:warning: The -With format parameter (and also -Paste & -Drop ) MUST be defined with single quotes. Using double quotes causes string interpolation to occur resulting in named group references to not be evaluated as expected. Let's re-run the last command, but using double quotes for the -With parameter:

:x: Rename-Many -Pattern '(?\<disc>\d{2})-' -Anchor '\d{2}' -With "\${_a}-${disc} -" -Top 10 -WhatIf

picture

This shows that '\${_a}' and '${disc}' are both evaluated to an empty string, breaking the desired result.

The final point worthy of note is the 'Undo Rename' in the summary. By default, all executed commands are undo-able (assuming the undo feature has not been disabled). If we find that after running the command (assuming it has been unlocked and -WhatIf is not specified), the results are not as envisioned (shouldn't really happen, because the -WhatIf should always be used for new executions), the rename can be undone.

The summary contains a path to an undo script under the 'Undo Rename' signal. The user can review its contents first (recommended before running any scripts on a system) and then source that file. The undo script is purely a sequence of renames in reverse with the original name and new names swapped around, thereby reversing the whole batch.

:gem: Move to Start

Move a regex match identified by -Pattern from the items' name from its current location to the start of an item's name.

Continuing with the audio files as discussed in Move To Anchor, let's say we want to move the \<TRACK-NO> to the start of an item's name.

Focusing on a single file item in the batch, this time being: '02-06 Airwaves.mp3':

:heavy_minus_sign: :one: Rename-Many -Pattern '\d{2}', 2 -Start -WhatIf

0602- Airwaves.mp3

However, as we discovered in the previous section, we need to do more to obtain a satisfactory result. We can tidy this up, with the use of the -With parameter:

:heavy_plus_sign: Rename-Many -Pattern '-(?\<track>\d{2})' -Start -With '${track}-' -Drop ' -' -WhatIf

06-02 - Airwaves.mp3

Let's explore each of the points that gets us to this result:

Now that we have our finalised version, lets see how this looks in a batch:

(actually, the screen shot below uses the Top parameter to reduce the number of items processed, for brevity)

picture

:warning: When using the -Start and -End anchors (this does not apply to Hybrid Anchors), the user should be aware that if the match is already at the target location, then it will be skipped. For example, if we had a series of directories that contained a date in its name, but the location of the date was inconsistent, we might decide we want to move the date for every directory to the end. However, some directories may already have the date at the end, so there is no point in processing these items. That is why some items may be skipped when using -Start and -End anchors.

:gem: Move to End

Move a regex match identified by -Pattern from the item's name from its current location to the end of its name.

This time, we want to move the \<DISK-NO> to the end of the item's name. The reader might be thinking well isn't this just the opposite to using Start? and they would be right. But in the discussion of Move To End we'll address some slightly different issues/techniques that illustrate other ways the command can be used.

Focusing on a single file item in the batch, this time being: '02-04 Intermission.mp3':

Our initial naive attempt might be:

:heavy_minus_sign: :two: Rename-Many -Pattern '\d{2}' -End -WhatIf

resulting in:

-04 Intermission02.mp3

This works, but it's not very graceful. So again we can optimise this via the -With formatter.

:heavy_minus_sign: Rename-Many -Pattern '(?\<disc>\d{2})-' -End -With ' (disc-${disc})' -WhatIf

04 Intermission (disc-02).mp3

We have chosen to spice up the formatter with extra content, but as we did before, we'll examine all the points that gets us to this state:

However, this is not our chosen solution. We want to insert a dash in between the TRACK-NO and TRACK-NAME as we did before so that our example is renamed to: '04 - Intermission (disc-02).mp3'

Achieving this, requires more work than we completed in our initial attempt :two:

:heavy_plus_sign: Rename-Many -Pattern '(?\<disc>\d{2})-(?\<track>\d{2})' -End -With ' (disc-\${disc})' -Drop '${track} -' -WhatIf

which results in:

04 - Intermission (disc-02).mp3

Bingo! The extra points worthy of note are:

Let's see this in our batch:

(actually, the screen shot below uses the Top parameter to reduce the number of items processed, for brevity)

picture

:gem: Move to Hybrid Anchor

In the rename batch, some items may match the -Anchor pattern and others may not. Ordinarily, if the -Anchor does not match, then the rename will not occur for this item. This would then require the user to re-run the command with a redefined anchor or run with an entirely different parameter set. However, with a Hybrid Anchor, what we're saying is:

If the specified anchor fails to match, then move the match to the Start or End

So using a hybrid anchor allows the user to perform an anchor oriented operation with a backup if the anchor doesn't match, all in the same batch.

Consider the following directory list in our somewhat contrived example:

picture

We've decided that we want to move the date (if it exists) to precede the fragment '- at'. So in this case the date is the target of the -Pattern match and '- at' is our -Anchor. The directories are not consistently named, so we'll encounter different results from each item.

Let's tailor our requirement a little and say, move the date to the -Anchor if it exists and if it doesn't, then move it to the end. This is where we need a Hybrid Anchor. The hybrid parameters are -AnchorStart and -AnchorEnd.

Before we start using a Hybrid Anchor let's see what happens when we use a regular one:

:heavy_minus_sign: Rename-Many -Pattern '(?\d{2}-\d{2}-\d{4})?' -Anchor '- at' -Relation 'before' -WhatIf

Results in (please excuse the wraparounds):

picture

Things to note:

So in this particular batch run, 2 items are not renamed because the -Anchor match failed. Now let's use a Hybrid Anchor (we simply replace -Anchor with -AnchorEnd):

:heavy_minus_sign: :three: Rename-Many -Pattern '\(?\d{2}-\d{2}-\d{4}\)?' -AnchorEnd '- at' -Relation 'before' -WhatIf

results in:

picture

Now, all the un-skipped items in the batch are renamed. When the Anchor does not match, the date is moved to the end.

But again, the date inserted could do with some alteration. Let's say we wanted to change the date format so that they are in the ISO form '\<YEAR>-\<MONTH>-\<DAY>'. To do this, our -Pattern has to make use of named capture groups again. Also, other literal text can be defined, which we can achieve by using the -With parameter.

But before we move on, let's take another look at our command in :three:. Did you notice anything different about this example? Well, since the directories contain characters that are special to regular expressions and we access them in our -Pattern, they need to be escaped. In this case, the open '(' and close ')' brackets need escaping hence the '\(' and '\)' in the -Pattern.

So, improving our command line, we get to:

:heavy_minus_sign: Rename-Many -Pattern '\(?(?\<d>\d{2})-(?\<m>\d{2})-(?\<y>\d{4})\)?' -AnchorEnd '- at' -With ' - on \${y}-\${m}-\${d} \${_a}' -WhatIf

Resulting in:

picture

Points of note:

On closer inspection, it appears we have an issue. Item 'Underworld - (01-06-1999) - at Brixton Academy' is renamed to: 'Underworld - - on 1999-06-01 - at Brixton Academy'. The '- -' looks ugly and is an un-intended result (caused by the leading ' -' previously mentioned), which occurs because of our -With replacement, not quite meeting the needs of this item. This kind of thing happens regularly and in this situation we can exclude it so that it can be processed in a separate batch. Since this is the only item to be excluded, we can use a very specific discriminating value for the -Except parameter, ie: 'Underworld':

:heavy_plus_sign: Rename-Many -Except 'Underworld' -Pattern '\(?(?\<d>\d{2})-(?\<m>\d{2})-(?\<y>\d{4})\)?' -AnchorEnd '- at' -With ' - on \${y}-\${m}-\${d} ${_a}' -WhatIf

and we finally arrive at:

picture

Typically, -Except would be just generic enough to single out items to be skipped, but can't be so general as to exclude items that ought not to be. We only need to exclude a single item in this case, so our regular expression can be as specific as it needs to be ('Underworld').

What we've learnt about -AnchorEnd hybrid applies identically to -AnchorStart, except that the -Pattern match is moved to the start.

:gem: Swap Content

The Drop facility has a beneficial side effect. It can also be used to simply swap 2 pieces of content over.

In this example we will use a new directory list and this time we have a list which is already normalised but we wish to swap over the positions of two fields. Consider the directory list:

picture

We can see that the directories are indeed structured uniformly, but we wish to swap over the date with the location. With the Drop facility, this can be achieved as follows:

:heavy_plus_sign: Rename-Many -Pattern '(?\\d{2}-\d{2}-\d{4})' -Anchor '- at (?\[\w\s]+)' -With '[\${date}]' -Drop '\${loc}'

and we arrive at:

picture

Points of note:

So in summary, the -Pattern and the -Anchor matches, comprise the two ends of the swap and we use -With and -Drop parameters respectively as the formatters whose contents reference the appropriate named group captures.

:sparkles: Update Match

Update Parameter Alias DESCRIPTION
Paste ps Formatter used when performing in place Update

Update-Match simply involves modifying a match in its present location. Since we don't have an -Anchor to deal with, it is much simpler to use than Move Match scenarios.

As well as -Anchor and related parameters, instead of using the -With parameter, we use the -Paste format parameter and it serves a similar purpose. The peculiarities of PowerShell parameter sets means that it is much easier to use a separate parameter, rather than to try and re-use -With in a different context (it is the same reason why new parameters were defined for the Hybrid Anchors, instead of re-purposing -Anchor/-Start/-End).

The file list which was the subject of Move To Hybrid will be used in the following discussion.

This time, we want to update the dates in place, changing the format to be in US date format (mm-dd-yyyy)

:heavy_plus_sign: Rename-Many -Pattern '\(?(?\<d>\d{2})-(?\<m>\d{2})-(?\<y>\d{4})\)?' -Paste '[\${m}-\${d}-\${y}]' -WhatIf

picture

:sparkles: Cut Match

Regex Parameter Alias DESCRIPTION
Cut :heavy_multiplication_x: Remove this match without a replacement

Simply removes the content matched by -Cut

Let's remove the date element from the directory list featured in the previous section

:heavy_plus_sign: Rename-Many -Cut '\(?(?\<d>\d{2})-(?\<m>\d{2})-(?\<y>\d{4})\)?' -WhatIf

picture

:sparkles: Add Appendage

Add Prefix

Prefix Parameter Alias DESCRIPTION
Prepend pr Prefix items' name with this literal string

Appends literal content to start of an item's name.

Using our audio file list, example from Move To Anchor, we can prefix each items name with some literal content:

:heavy_plus_sign: Rename-Many -Prepend 'Kraftwerk - ' -Top 10 -WhatIf

picture

Add Suffix

Suffix Parameter Alias DESCRIPTION
Append ap Append this literal string to items' name

Appends literal content to end of an item's name.

Eg:

:heavy_plus_sign: Rename-Many -Append ' - Kraftwerk' -Top 10 -WhatIf

:sparkles: Parameter Reference

:dart: Anchor

Type: [array](regular expression, string)

Indicates that the rename operation will be a move of the token from its original point to the point indicated by -Anchor. -Anchor is a regular expression string applied to the pipeline item's name (after the -Pattern match has been removed). The -Pattern match that is removed is inserted at the position indicated by the anchor match in collaboration with the Relation parameter.

:dart: AnchorEnd

Type: [array](regular expression, string)

Similar to -Anchor except that if the pattern specified by -AnchorEnd does not match, then the -Pattern match will be moved to the end. This is known as a Hybrid Anchor.

:dart: AnchorStart

Type: [array](regular expression, string)

Similar to -Anchor except that if the pattern specified by -AnchorStart does not match, then the -Pattern match will be moved to the start. This is known as a Hybrid Anchor.

:dart: Append

Type: [string]

Appends a literal string to end of item's name.

:dart: Condition

Type: [ScriptBlock] (predicate)

Provides another way of filtering pipeline items. This is not typically specified on the command line, rather it is meant for those wanting to build functionality on top of Rename-Many.

:dart: Context

Type: [PSCustomObject]

Provides another way of customising Rename-Many. This is not typically specified on the command line, rather it is meant for those wanting to build functionality on top of Rename-Many. Context should be a PSCustomObject with the following note properties:

:dart: Copy

Type: [array](regular expression, string)

Regular expression string applied to the pipeline item's name (after the -Pattern match has been removed), indicating a portion which should be copied and re-inserted (via the format parameters -Paste and -With). Since this is a regular expression to be used in -Paste/-With, there is no value in the user specifying a static pattern, because that literal string can just be defined in -Paste/-With. The value in the -Copy parameter comes when a generic pattern is defined eg \d{3} (is non Literal), specifies any 3 digits as opposed to say '123', which could be used directly in the formatter parameters without the need for -Copy. The match defined by -Copy is stored in special variable ${_c} and can be referenced as such from -Paste and -With.

:dart: Cut

Type: [array](regular expression, string)

Is a replacement for the -Pattern parameter, when a Cut operation is required. The pattern match will be removed from the item's name and no other replacement occurs.

:dart: Diagnose

Type: [switch]

Indicates the command should be run in -WhatIf mode. When enabled, it presents additional information that assists the user in correcting the un-expected results caused by an incorrect/un-intended regular expression. The current diagnosis will show the contents of named capture groups that they may have specified. When an item is not renamed (usually because of an incorrect regular expression), the user can use the diagnostics along side the 'Not Renamed' reason to track down errors. When -Diagnose has been specified, -WhatIf does not need to be specified.

:dart: Directory

Type: [switch]

Indicates only Directory items in the pipeline will be processed. If neither this switch or the File switch are specified, then both File and Directory items are processed.

:dart: Drop

Type: [string]

Only applicable to move operations. Defines what text is used to replace the -Pattern match with. So in this use-case, the user wants to move a particular token/pattern to another part of the name and at the same time drop content in the location -Pattern was removed from.

See also Swap Content

:dart: End

Type: [switch]

Is another type of anchor used instead of -Anchor and specifies that the -Pattern match should be moved to the end of the new name.

:dart: Except

Type: [string](regular expression)

Regular expression string applied to the original pipeline item's name (before the -Pattern match has been removed). Allows the user to exclude some items that have been fed in via the pipeline. Those items that match the exclusion are skipped during the rename batch.

:dart: File

Type: [switch]

Indicates only File items in the pipeline will be processed. If neither this switch or the -Directory switch are specified, then both File and Directory items are processed.

:dart: Include

Type: [string](regular expression)

Regular expression string applied to the original pipeline item's name (before the -Pattern match has been removed). Allows the user to include some items that have been fed in via the pipeline. Only those items that match -Include pattern are included during the rename batch, the others are skipped. The value of the -Include parameter comes when you want to define a pattern which pipeline items match, without it be removed from the original name, which is what happens with -Pattern. Eg, the user may want to specify the only items that should be considered a candidate to be renamed are those that match a particular pattern but doing so in -Pattern would simply remove that pattern. That may be ok, but if it's not, the user should specify a pattern in the -Include and use -Pattern for the match you do want to be moved (with the anchor parameters) or replaced (with the formatter parameters).

:dart: Paste

Type: [string]

Formatter parameter for Update operations. Can contain named/numbered group references defined inside regular expression parameters, or use special named references \$0 for the whole -Pattern match and ${_c} for the whole -Copy match.

:dart: Pattern

Type: [array](regular expression, string)

Regular expression string that indicates which part of the pipeline items' name that either needs to be moved or replaced as part of bulk rename operation. Those characters in the name which match are removed from the name.

:dart: Prepend

Type: [string]

Prefixes a literal string to start of item's name.

:dart: Relation

Type: [string]("before" | "after")

Used in conjunction with the -Anchor parameter and can be set to either 'before' or 'after' (the default). Defines the relationship of the -Pattern match with the -Anchor match in the new name for the pipeline item.

:dart: Start

Type: [switch]

Another type of anchor used instead of -Anchor and specifies that the -Pattern match should be moved to the start of the new name.

:dart: Test

Type: [switch]

Indicates if this is being invoked from a test case, so that the output can be suppressed if appropriate. By default, the test cases should be quiet. During development and test stage, the user might want to see actual output in the console. The presence of variable 'EliziumTest' in the environment will enable verbose tests. When invoked by an interactive user in production environment, the Test flag should not be set. Doing so will suppress the output depending on the presence of 'EliziumTest'. ALL test cases should specify this Test flag.

:dart: Top

Type: [int]

A number indicating how many items to process. If it is known that the number of items that will be candidates to be renamed is large, the user can limit this to the first -Top number of items. This is typically used as an exploratory tool, to determine the effects of the rename operation.

:dart: Transform

Type: [ScriptBlock]

A script block which is given the chance to perform a modification to the finally named item. The transform is invoked prior to post-processing, so that the post-processing rules are not breached and the transform does not have to worry about breaking them. The transform function's signature is as follows:

and should return the new name. If the transform does not change the name, it should return an empty string.

:dart: underscore

Type: [FileSystemInfo]

The pipeline item which should either be an instance of FileInfo or DirectoryInfo.

:dart: Whole

Type: [char]

Provides an alternative way to indicate that the regular expression parameters should be treated as a whole word (it just wraps the expression inside \b tokens). If set to '*', then it applies to all expression parameters otherwise a single letter can specify which of the parameters 'Whole' should be applied to. Valid values are:

:dart: With

Type: [string]

Formatter which defines what text is used as the replacement for the -Pattern match. Works in concert with -Relation (whereas -Paste does not). -With can reference special variables:

When -Pattern contains named capture groups, these variables can also be referenced. Eg if the -Pattern is defined as '(?\\d{1,2})-(?\\d{1,2})-(?\\d{4})', then the variables \${day}, \${mon} and \${year} also become available for use in -With or -Paste.

Typically, -With is literal text which is used to replace the -Pattern match and is inserted according to the anchor parameters and -Relation. When using -With, whatever is defined in the anchor match IS removed from the pipeline's name and requires the user to re-insert it with '${_a}' inside -With if so required. The reason for this is that when -With is not present, the -Pattern match content is inserted verbatim next to the -Anchor either before or after. But if we use a -With, then the user has full control over whereabouts the -Anchor is inserted inside -With, so -Relation is redundant.

:radioactive: Troubleshooting / Common Errors

Most issues that occur with using Rename-Many are the result of not defining a regex pattern correctly. These clearly can only be fixed, by reviewing the pattern and adjusting accordingly.

It is advised that users always run with -WhatIf enabled for new invocations of Rename-Many, so that the results can be confirmed before being actioned for real. When unexpected results occur, the user can specify the -Diagnose parameter to ensure that named capture groups are working as expected. Readers are also directed to use other 3rd party resources to debug regex patterns such as regex101.

Common issues include:

:hammer: Expanding Rename-Many Capabilities

:mortar_board: Higher Order Commands

Provides the facility to reuse the base Rename-Many functionality to build higher level functionality. Since regular expressions are not particularly easy to specify without prior skill and knowledge, it might be advantageous to build a high level command that wraps the base functionality and has embedded within it a commonly used combination of parameters and regular expression definitions. This way the high level version can hide-away difficult to remember regular expressions that are used on a regular basis.

To build a high level Rename-Many command, a developer should perform the following tasks:

Let's say we want to create a command to re-arrange dates in UK format (dd-mm-yyyy) to ISO format 'yyyy-mm-dd', called Convert-Date

:pencil: 1) Define a context

  [PSCustomObject]$Context = [PSCustomObject]@{
    Title             = 'Reformat UK dates to ISO Format';
    ItemMessage       = 'To ISO format';
    SummaryMessage    = 'UK Dates Converted to ISO';
    Locked            = 'CONVERT_UK_DATES_LOCKED';
    UndoDisabledEnVar = 'CONVERT_UK_DATES_UNDO_DISABLED';
    OperantShortCode  = 'convuk';
  }

Taking one of the previously displayed screen-shots ...:

picture

... we can see, where some of those context items appear in the output. From this screen-shot:

The other non UI elements:

:pencil: 2) Define the arguments passed into 'Rename-Many'

These arguments will be hardcoded into Convert-Date, so that the end user doesn't have to specify them.

:pencil: 3) Define new user facing parameters

We can either accept file system objects from the pipeline (but they would need to be collected up and passed into a new pipeline involving Rename-Many) or define a new -Path like parameter. We'll do the latter in this case and call it -LiteralPath.

We probably need to replicate -WhatIf and -Diagnose, which would be forwarded onto Rename-Many. In the case of -WhatIf, we don't define that explicitly, instead we decorate our function with SupportsShouldProcess. But it should be noted that generally, the value of -WhatIf flows from the user invoked command to other standard PowerShell functions (eg Move-Item), but it doesn't flow from one 3rd party command to another unless they are in the same module. That is to say, -WhatIf does not cross module boundaries (except the standard PowerShell functions). This means we need to forward the value of -WhatIf explicitly, by doing something like:

-WhatIf:$($PSBoundParameters.ContainsKey('WhatIf'))

Since $WhatIf doesn't exist in its own right, we need to use $PSBoundParameters to see if it is present in the bound parameters.

Now we end up with a command signature as follows:

  function Convert-Date {
    [Diagnostics.CodeAnalysis.SuppressMessageAttribute('PSShouldProcess', '')]
    [CmdletBinding(SupportsShouldProcess)]
    [Alias('convuk')]
    param( 
      [Parameter()]
      [string]$LiteralPath,

      [Parameter()]
      [switch]$Diagnose
    )
    ...
  }

:pencil: 4) Define the implementation

Typically, the parameters to Rename-Many would be splatted, via a hashtable.

  function Convert-Date {
    # signature not repeated, see previous code snippet instead

    [PSCustomObject]$context = @{
      Title             = 'Reformat UK dates to ISO Format';
      ItemMessage       = 'To ISO format';
      SummaryMessage    = 'UK Dates Converted to ISO';
      Locked            = 'CONVERT_UK_DATES_LOCKED';
      UndoDisabledEnVar = 'CONVERT_UK_DATES_UNDO_DISABLED';
      OperantShortCode  = 'convuk';
    }

    [hashtable]$parameters = @{
      'Pattern'  = '(?<d>\d{2})-(?<m>\d{2})-(?<y>\d{4})';
      'Paste'    = '(${y}-${m}-${d})';
      'Context'  = $context;
      'Diagnose' = $Diagnose.IsPresent;
      'WhatIf'   = $PSBoundParameters.ContainsKey('WhatIf');
    }

    Get-ChildItem -LiteralPath $LiteralPath | Rename-Many @parameters
  }

The user can use the higher order command Convert-Date with a simpler more specialised interface instead of Rename-Many, eg:

:heavy_plus_sign: Convert-Date -LiteralPath ~/logs -WhatIf

Note, there is another way of complementing the functionality of existing commands that would apply in this scenario and that is with a Proxy Command. This is a slightly more involved process but contains some of the same techniques just discussed. It's out of the scope of this documentation, but here is a blog post that describes how to apply this technique.

:robot: Using Transform

Another way to obtain custom functionality from Rename-Many is to provide a custom script-block for the Transform parameter. This will be illustrated with an example that replaces all '[' with '(' and all ')' with ']'. This might be useful to skirt round a widely encountered problem using the -Path parameter of mutating commands like Rename-Item which ascribes custom semantics to '[' and ']' in file system paths, causing un-expected results. (This can be averted using -LiteralPath, but this is just an example to discuss using the Transform parameter.)

The signature of the Transform script-block is as follows

  param(
    [Parameter()]
    [string]$Original,

    [Parameter()]
    [string]$Renamed,

    [Parameter()]
    [string]$PatternCapture,

    [Parameter()]
    [hashtable]$Exchange
  )

where:

This could be implemented as follows

  [ScriptBlock]$transformer = [ScriptBlock] {
    param($Original, $Renamed, $PatternCapture, $Exchange)
    return $Original.Replace('[', '(').Replace(']', ')');
  }

... and theoretically, could be invoked as:

:heavy_multiplication_x: Rename-Many -Pattern '\[[\w]+\]' -Transform $transformer -WhatIf

But, the Transform parameter is not meant to be used interactively (although it could be, but would be cumbersome). Rather, the intention is that the user would create a Higher Order Command so invoking this function interactively would become more convenient on the command line.

:pushpin: In this example, the -Pattern becomes a filter, such that only items containing any word characters (\w) inside square brackets are processed.

:green_salad: Recipes

This section will contain some examples that solve common renaming requirements. They are intended to give the reader a jump start in defining their own regular expressions and formatters to use with Rename-Many.

:hammer: Developer Notes

This module has the following developer dependencies:

After cloning the repo, change to the Elizium.RexFs directory from the root. You can look at the build script Elizium.RexFs.build.ps1, it will contain various tasks, the most important of which are explained below

Running build tasks

To build the module and run the unit tests:

invoke-build

To build the module only:

invoke-build build

To Run the unit tests only (assuming already built)

invoke-build tests

To build external help:

invoke-build buildHelp

Problem rebuilding modified classes in the same PowerShell session

:warning: Elizium.RexFs makes use of PowerShell classes. Because of the nature of classes in PowerShell, re-building edited code can cause errors. This is not a fault of the Elizium.RexFs code, it's just the way PowerShell classes have been designed.

What you will find is, if a class has been modified then rebuilt in the same session, you may find multiple class errors like so:

[-] EndAdapter.given: EndAdapter.should: get name 31ms (30ms|1ms)
 PSInvalidCastException: Cannot convert the "EndAdapter" value of type "EndAdapter" to type "EndAdapter".
 ArgumentTransformationMetadataException: Cannot convert the "EndAdapter" value of type "EndAdapter" to type "EndAdapter".
 at <ScriptBlock>, ..\github\PoSh\RexFs\Elizium.RexFs\Tests\Rename-Many.tests.ps1:21

Fear not, this is just reporting that the class definition has changed and because of this difference, one can't be substituted for another in the same PowerShell session (this is in contrast to the way functions work, where you can simply re-define a function in the same session and it will replace the previous definition. This luxury has not been afforded to classes unfortunately). All that's required is to restart a new session. The rebuild in the new session should progress without these errors.

It is a bit onerous having to restart a session for every build, but below is a function that can be defined in the users powershell profile that when invoked, begins a restart loop. Now, when an exit is issued, the session is automatically restarted:

Helper function restart-session

Insert this into your PowerShell session file.

function Get-TagPath {
  return Join-Path $env:temp -ChildPath 'restart-session.tag.txt';
}

function Restart-Session {
  [Alias('ress')]
  param()

  [string]$tagPath = Get-TagPath;
  if (-not([string]::IsNullOrEmpty($env:tag))) {
    Set-Content -Path $tagPath -Value $env:tag;
  }
  elseif (Test-Path -Path $tagPath) {
    Remove-Item -Path $tagPath;
  }

  [System.Management.Automation.PathInfo]$pathInfo = Get-Location;
  while ($true) {
    pwsh -Command {
      [string]$tagPath = Get-TagPath;
      [string]$greeting = "🍺 Restarted!";
      if (Test-Path -Path $tagPath) {
        $tag = Get-Content -Path $tagPath;

        if (($tag -is [string]) -or ($tag -is [string[]])) {
          $env:tag = $tag;
          $greeting = "🍺 Restarted! (Pester Tag: '$env:tag' ✔️)";
        }
      }

      Write-Host -ForegroundColor 'Cyan' $greeting;
    } -NoExit -WorkingDirectory $($pathInfo.Path)
    if ($LASTEXITCODE) {
      break
    }
  }
}

Another feature this function possesses is the restoration of the Tag environment variable. The Tag is used to control which testcases Pester runs. Pester contains a Tag in its configuration and when set, it will only run those test cases decorated with this tag value.

So, when a restart occurs, the Tag if set is restored and you will see which tag is in play as part of the restart. If no tag is found then no tag is restored. This function just helps the tedium of having to keep redefining the Tag in-between restarts, as now this is automatically restored.

The sequence goes:

After restart, tag is restored and the restart message will indicate as such

... and repeat.

:triangular_flag_on_post: EliziumTest flag

The user can set this flag in the environment (just set it to any non $null value).

By default, $env:EliziumTest, will not be present, this means, that the unit tests in Elizium.RexFs will run in silent mode. However, there are some tests which are less valuable in silent mode, doing so would invalidate them to some degree. There are only a few of the tests in this category (tagged as 'Host') and it's because they require Write-Host to be invoked. Theoretically, one could mock out the Write-Host call, but some errors can be much easier to spot visually. This generally is not the best technique in unit-testing, but these test cases have been backed up by non noisy equivalents to make sure all bases are covered.

During development, it is very useful to get a visual on how ui commands are behaving. This was the rationale behind the introduction of this flag. So when EliziumTest is defined, the user will see more output that reflects the execution of the Scribbler and Krayon.