Buildgen doesn't work on Spindle?

[omerzach]

It looks like buildgen in fsqio doesn't seem to be updating dependencies in src/thrift:

Omers-MacBook-Pro:fsqio omer$ git status
On branch master
Your branch is up-to-date with 'origin/master'.

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

    modified:   src/thrift/io/fsq/twofishes/BUILD

no changes added to commit (use "git add" and/or "git commit -a")
Omers-MacBook-Pro:fsqio omer$ git diff src/thrift/io/fsq/twofishes/BUILD
diff --git a/src/thrift/io/fsq/twofishes/BUILD b/src/thrift/io/fsq/twofishes/BUILD
index 2846792..dbcaa37 100644
--- a/src/thrift/io/fsq/twofishes/BUILD
+++ b/src/thrift/io/fsq/twofishes/BUILD
@@ -3,9 +3,6 @@
 scala_record_library(
   name = 'twofishes',
   dependencies = [
-    '3rdparty:bson',
-    '3rdparty:twitter-util',
-    'src/jvm/io/fsq/spindle/runtime',
   ],
   sources = globs('*.thrift'),
   provides=scala_artifact(
Omers-MacBook-Pro:fsqio omer$ ./pants buildgen

22:31:09 00:00 [main]
               (To run a reporting server: ./pants server)
22:31:09 00:00   [setup]
22:31:09 00:00     [parse]
               Executing tasks in goals: tag -> bootstrap -> imports -> unpack-jars -> validate -> deferred-sources -> build-spindle -> jvm-platform-validate -> gen -> webpack -> map-java-exported-symbols -> map-scala-exported-symbols -> map-jvm-symbol-to-source-tree -> resolve -> map-third-party-jar-symbols -> map-scala-used-symbols -> map-sources-to-addresses-mapper -> map-scala-library-used-addresses -> map-python-exported-symbols -> map-derived-targets -> buildgen
22:31:10 00:01   [tag]
22:31:10 00:01     [tag]
22:31:10 00:01   [bootstrap]
22:31:10 00:01     [substitute-aliased-targets]
22:31:10 00:01     [bootstrap-jvm-tools]
22:31:10 00:01     [provide-tools-jar]
22:31:10 00:01     [global-jar-dependency-management]
22:31:10 00:01   [imports]
22:31:10 00:01     [ivy-imports]
22:31:10 00:01   [unpack-jars]
22:31:11 00:02     [unpack-jars]
22:31:11 00:02   [validate]
22:31:11 00:02     [validate]
22:31:11 00:02   [deferred-sources]
22:31:11 00:02     [deferred-sources]
22:31:11 00:02   [build-spindle]
22:31:11 00:02     [build-spindle]
22:31:11 00:02   [jvm-platform-validate]
22:31:11 00:02     [jvm-platform-validate]
22:31:11 00:02   [gen]
22:31:11 00:02     [antlr-java]
22:31:11 00:02     [antlr-py]
22:31:11 00:02     [jaxb]
22:31:11 00:02     [protoc]
22:31:11 00:02     [ragel]
22:31:11 00:02     [thrift-java]
22:31:11 00:02     [thrift-py]
22:31:11 00:02     [wire]
22:31:11 00:02     [go-thrift]
22:31:11 00:02     [spindle]
22:31:11 00:02     [validate-graph]
22:31:11 00:02   [webpack]
22:31:11 00:02     [webpack-resolve]
22:31:11 00:02     [webpack-gen]
22:31:12 00:03   [map-java-exported-symbols]
22:31:12 00:03     [map-java-exported-symbols]
22:31:12 00:03   [map-scala-exported-symbols]
22:31:12 00:03     [map-scala-exported-symbols]
22:31:12 00:03   [map-jvm-symbol-to-source-tree]
22:31:12 00:03     [map-jvm-symbol-to-source-tree]
22:31:12 00:03   [resolve]
22:31:12 00:03     [ivy]
22:31:13 00:04     [go]
22:31:13 00:04   [map-third-party-jar-symbols]
22:31:13 00:04     [map-third-party-jar-symbols]
22:31:13 00:04   [map-scala-used-symbols]
22:31:13 00:04     [map-scala-used-symbols]
22:31:13 00:04   [map-sources-to-addresses-mapper]
22:31:13 00:04     [map-sources-to-addresses-mapper]
22:31:13 00:04   [map-scala-library-used-addresses]
22:31:13 00:04     [map-scala-library-used-addresses]
22:31:13 00:04   [map-python-exported-symbols]
22:31:13 00:04     [map-python-exported-symbols]
22:31:13 00:04   [map-derived-targets]
22:31:13 00:04     [map-derived-targets]
22:31:13 00:04   [buildgen]
22:31:13 00:04     [go]
22:31:13 00:04     [buildgen]
22:31:13 00:04     [aggregate-targets]
22:31:13 00:04     [scala]WARN] BuildFileManipulator would have added test/thrift/io/fsq/spindle/codegen/parser/test:test as a dependency of test/jvm/io/fsq/spindle/codegen/binary/test:test, but that dependency was already forced with a comment.

22:31:13 00:04     [python]
22:31:14 00:05   [complete]
               SUCCESS

I recall it working there and see there's some code in buildgen referring to Spindle.

[mateor]

Actually, this is a bug from when I refactored the buildgen backend for publishing. I fixed this internally and just need to update. But in the meantime, I think it will work if you just add the backend to line 18 of pants.ini.

+    'fsqio.pants.buildgen

[omerzach]

Ah yep that works for fsqio, but not in my own repo.

I had

backend_packages: +[
    'fsqio.pants.spindle', # inlined
    'pants.contrib.buildgen.core',
    'pants.contrib.buildgen.jvm',
  ]

Adding fsqio.pants.buildgen yields

Omers-MacBook-Pro:sigma-monorepo omer$ ./pants buildgen
Traceback (most recent call last):
  File "/Users/omer/.cache/pants/setup/bootstrap-Darwin-x86_64/1.2.1/lib/python2.7/site-packages/pants/bin/extension_loader.py", line 123, in load_backend
    module = importlib.import_module(backend_module)
  File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
ImportError: No module named buildgen.register
Exception caught: (<class 'pants.base.exceptions.BackendConfigurationError'>)
  File "/Users/omer/.cache/pants/setup/bootstrap-Darwin-x86_64/1.2.1/bin/pants", line 11, in <module>
    sys.exit(main())
  File "/Users/omer/.cache/pants/setup/bootstrap-Darwin-x86_64/1.2.1/lib/python2.7/site-packages/pants/bin/pants_exe.py", line 44, in main
    PantsRunner(exiter).run()
  File "/Users/omer/.cache/pants/setup/bootstrap-Darwin-x86_64/1.2.1/lib/python2.7/site-packages/pants/bin/pants_runner.py", line 57, in run
    options_bootstrapper=options_bootstrapper)
  File "/Users/omer/.cache/pants/setup/bootstrap-Darwin-x86_64/1.2.1/lib/python2.7/site-packages/pants/bin/pants_runner.py", line 46, in _run
    return LocalPantsRunner(exiter, args, env, options_bootstrapper=options_bootstrapper).run()
  File "/Users/omer/.cache/pants/setup/bootstrap-Darwin-x86_64/1.2.1/lib/python2.7/site-packages/pants/bin/local_pants_runner.py", line 53, in run
    self._maybe_profiled(self._run)
  File "/Users/omer/.cache/pants/setup/bootstrap-Darwin-x86_64/1.2.1/lib/python2.7/site-packages/pants/bin/local_pants_runner.py", line 50, in _maybe_profiled
    runner()
  File "/Users/omer/.cache/pants/setup/bootstrap-Darwin-x86_64/1.2.1/lib/python2.7/site-packages/pants/bin/local_pants_runner.py", line 59, in _run
    options, build_config = OptionsInitializer(options_bootstrapper, exiter=self._exiter).setup()
  File "/Users/omer/.cache/pants/setup/bootstrap-Darwin-x86_64/1.2.1/lib/python2.7/site-packages/pants/bin/options_initializer.py", line 174, in setup
    backends)
  File "/Users/omer/.cache/pants/setup/bootstrap-Darwin-x86_64/1.2.1/lib/python2.7/site-packages/pants/bin/options_initializer.py", line 83, in _load_plugins
    return load_backends_and_plugins(plugins, working_set, backend_packages)
  File "/Users/omer/.cache/pants/setup/bootstrap-Darwin-x86_64/1.2.1/lib/python2.7/site-packages/pants/bin/extension_loader.py", line 36, in load_backends_and_plugins
    load_build_configuration_from_source(build_configuration, backends)
  File "/Users/omer/.cache/pants/setup/bootstrap-Darwin-x86_64/1.2.1/lib/python2.7/site-packages/pants/bin/extension_loader.py", line 109, in load_build_configuration_from_source
    load_backend(build_configuration, backend_package)
  File "/Users/omer/.cache/pants/setup/bootstrap-Darwin-x86_64/1.2.1/lib/python2.7/site-packages/pants/bin/extension_loader.py", line 127, in load_backend
    .format(backend=backend_module, error=e))

Exception message: Failed to load the fsqio.pants.buildgen.register backend: No module named buildgen.register

Is there a way to get fsqio.pants.buildgen to work other than inlining the code from fsqio?

[mateor]

Ah - so we would need to add a new package for spindle buildgen.

What version of Pants and buildgen plugins are you using?

[mateor]

Publishing new packages to PyPi is easy, for the record, we just didn't have that code pushed to open source yet. I think that you could update the plugins independent of Pants versions but it might not hold all the way back to 1.2.1.

The buildgen_spindle is a one file task though, so if you couldn't update the plugins, you would only have to inline the spindle code as a tiny plugin.

[omerzach]

We're on pants_version: 1.2.1 and only using it for Scala (so core/jvm, no python)

[mateor]

So - I guess the question is up to you. If you want buildgen_spindle, I am happy to publish that as a module to pypi.

But, our buildgen modules are published to work with Pants 1.2.0 and greater and how well that works in practice is an open question. You may find that adding the loose source is easier for you.

I intend to update Fsq.io sometime this week, I can publish a spindle_buildgen module without much effort and then you can let me know

[omrihq]

Any update on this?

[omerzach]

@mateor: Yeah if you can publish a spindle_buildgen module we'd like to give that a shot (@omrihq works with me at Sigma and is blocked on something related too)

[mateor]

👋

Yeah, I forgot honestly, although I thought I had gotten enough of the general info in that thread, all that was missing was the exact current package name.

Anyway, it was due. I updated fsqio tonight and also configured a PyPi module for fsqio.pants.buildgen.spindle.

If you check out Fsq.io master, you should be able to fully demo consuming buildgen from hosted repo with the following small diff

mateo-2:fsqio mateo$ git diff --cached
diff --git a/BUILD.tools b/BUILD.tools
new file mode 100644
index 00000000..c9980895
--- /dev/null
+++ b/BUILD.tools
@@ -0,0 +1,15 @@
+
+jar_library(
+  name = 'buildgen-emit-exported-symbols',
+  jars = [
+    scala_jar(org = 'io.fsq', name = 'buildgen-emit-exported-symbols', rev = '1.2.0'),
+  ],
+)
+
+
+jar_library(
+  name = 'buildgen-emit-used-symbols',
+  jars = [
+    scala_jar(org = 'io.fsq', name = 'buildgen-emit-used-symbols', rev = '1.2.0'),
+  ],
+)
\ No newline at end of file
diff --git a/pants.ini b/pants.ini
index 50241c8a..332325c1 100644
--- a/pants.ini
+++ b/pants.ini
@@ -12,6 +12,13 @@ pants_version: 1.3.1rc1
 verify_config: True
 kill_nailguns: True

+plugins: [
+    "fsqio.pants.buildgen.core==1.3.1",
+    "fsqio.pants.buildgen.jvm==1.3.1",
+    "fsqio.pants.buildgen.python==1.3.1",
+    "fsqio.pants.buildgen.spindle==1.3.1",
+  ]
+
 backend_packages: +[
     'pants.contrib.go',
     'pants.contrib.node',

[mateor]

How I tested that in Fsqio.

1. git pull fsqio master
2. Add the above pants.ini change (it already has the BUILD.tools part in the BUILD.opensource file)
3. Delete some dependencies from scala_record_library files:

4. Changed some scala_record_library and buildgen_bag BUILD files and run buildgen

   modified:   src/thrift/io/fsq/BUILD
   modified:   src/thrift/io/fsq/common/thrift/descriptors/services/gen/BUILD

   $ git pull fsqio master $ rm -rf src/python/fsqio/pants/buildgen $ rm -rf test/python/fsqio_test/pants/buildgen $ rm -rf ~/.cache/fsqio $ ./pants buildgen