Examples of custom transformations?

[yang]

From reading the docs, we have a setup that performs various simple copies, moves, etc.

Are there examples of more complex transformations?

Such as:

• Any subdirectory named .copybara/ gets its files copied out into the parent dir, so you can have .copybara/ scattered throughout the codebase and colocated with the overlaid files.
• More custom munging of files with arbitrary code (for instance, parsing ASTs using existing language parsing libraries/tools).

[mikelalcon]

You can use dynamic transformations like:

def do_something(ctx):
    for f in ctx.run(glob(["**/.copybara/**"])):
        ctx.run(core.move(f.path, f.path.replace(".copybara/","")))      

core.workflow(
  ...
  transformations = [..., do_something, ...]
  ...
)

Regarding AST, sadly we don't have support for AST manipulation. We would like to add it in the future.

[yang]

Hi @mikelalcon , this seems to run extremely slowly, to the point where we've never seen copybara terminate - is this expected?

Before, we had a transformation of just core.move("app/.copybara", "app", overwrite=True).

The step would finish instantly:

0915 22:03:44.445 TASK: Cleaning output directory
0915 22:03:44.518 TASK: Running migrate
0915 22:03:44.594 TASK: Loading config copy.bara.sky
0915 22:03:45.364 TASK: Validating configuration
0915 22:03:45.373 TASK: Getting last revision: Resolving origin reference
0915 22:03:45.376 TASK: Git Origin: Initializing local repo
0915 22:05:27.220 TASK: Git Destination: Fetching: file:///public refs/heads/master
0915 22:05:43.015 TASK: Cleaning working directory
0915 22:05:43.030 TASK: Checking out the change
0915 22:05:45.903 TASK: Removing excluded origin files
0915 22:05:46.871 INFO: Removed 876 files from workdir that do not match origin_files
0915 22:05:47.139 TASK: [ 1/2] Transform Moving
0915 22:05:47.141 TASK: Moving
0915 22:05:47.216 TASK: [ 2/2] Transform Moving app/.copybara
0915 22:05:47.218 TASK: Moving app/.copybara
0915 22:05:47.244 TASK: Checking that destination_files covers all files in transform result
0915 22:05:47.632 TASK: Git Destination: Checking out master
0915 22:05:52.081 TASK: Git Destination: Adding all files
0915 22:05:58.128 TASK: Git Destination: Excluding files
0915 22:05:58.236 TASK: Git Destination: Creating a local commit
0915 22:06:01.222 TASK: Git Destination: Pushing to file:///public refs/heads/master

Now, we're using something like what you suggested:

def overlay_external_files(ctx):
    for f in ctx.run(glob(["**/*.external*"])):
        ctx.run(core.move(f.path, f.path.replace(".external",""), overwrite=True))

The step now has not finished running (has been running for a few hours):

0915 22:07:39.462 TASK: Cleaning output directory
0915 22:07:39.543 TASK: Running migrate
0915 22:07:39.606 TASK: Loading config copy.bara.sky
0915 22:07:40.395 TASK: Validating configuration
0915 22:07:40.404 TASK: Getting last revision: Resolving origin reference
0915 22:07:40.407 TASK: Git Origin: Initializing local repo
0915 22:09:23.649 TASK: Git Destination: Fetching: file:///public refs/heads/master
0915 22:09:38.879 WARN: '74cc45b37f1c635cca06a5594c046ae25a6d1e49' has been already migrated. Migrating anyway because of --force
0915 22:09:38.902 WARN: No changes from 74cc45b37f1c635cca06a5594c046ae25a6d1e49 up to 74cc45b37f1c635cca06a5594c046ae25a6d1e49 match any origin_files. Migrating anyway because of --force
0915 22:09:38.915 TASK: Cleaning working directory
0915 22:09:38.930 TASK: Checking out the change
0915 22:09:41.648 TASK: Removing excluded origin files
0915 22:09:42.563 INFO: Removed 876 files from workdir that do not match origin_files
0915 22:09:43.119 TASK: [ 1/2] Transform Moving
0915 22:09:43.122 TASK: Moving
0915 22:09:43.250 TASK: [ 2/2] Transform overlay_external_files

CPU is definitely working, just not sure what's going on.

Is there a way to speed this up?

The whole config is just:

core.workflow(
    name = "private_to_public",
    origin = git.origin(
        url = privateLocalUrl,
        ref = "master",
    ),
    destination = git.destination(
        url = publicLocalUrl,
        fetch = "master",
        push = "master",
    ),
    origin_files = globs_in_private,
    destination_files = glob(["app/**"]),
    authoring = authoring.pass_thru("Copybara <ops+copybara@example.com>"),
    # Must disable this for the first-time run, since everything should be squashed into one commit (you'll get some kind of error from copybara)
    # mode = 'ITERATIVE',
    transformations = [
        # Move all files into app/ subdir.
        core.move("", "app"),
        # core.move("app/.copybara", "app", overwrite=True),

        # Overlay all the files named *.external* - these are the public versions of these files.
        overlay_external_files,
    ],
)