search

qitab / cl-protobufs

Common Lisp protocol buffer implementation.
MIT License
83 stars 17 forks source link

Unable to compile opentelemetry-proto through asdf #431

Closed michaeldelago closed 1 week ago

michaeldelago commented 7 months ago

At the moment, I'm unable to compile the proto files for opentelemetry. I have an asd like so:

(defsystem "opentelemetry-cl"
  :version "0.1.0"
  :depends-on (:cl-protobufs)
  :defsystem-depends-on (:cl-protobufs.asdf)
  :components ((:module "opentelemetry-proto"
                :serial t
                :components
                ((:protobuf-source-file "common"
                  :proto-pathname "opentelemetry/proto/common/v1/common.proto"
                  :proto-search-path ("./"))
                 (:protobuf-source-file "resource"
                  :proto-pathname "opentelemetry/proto/resource/v1/resource.proto"
                  :depends-on ("common")
                  :proto-search-path ("./"))))))

opentelemetry-proto is cloned to a subdirectory, which can be done with this command:

$ git clone --branch=v1.1.0 https://github.com/open-telemetry/opentelemetry-proto opentelemetry-proto

I'm using qlot as well:

$ qlot add qitab/cl-protobufs
$ qlot install

When attempting to load the system, I receive an error the following error message:

$ qlot exec rlwrap ros run -s opentelemetry-cl

debugger invoked on a CL-PROTOBUFS:PROTOBUF-ERROR in thread
#<THREAD tid=1273506 "main thread" RUNNING {10013E0003}>:
  Could not find file "opentelemetry/proto/common/v1/common.proto" imported by #<CL-PROTOBUFS:FILE-DESCRIPTOR RESOURCE (package opente
lemetry.proto.resource.v1) {1004AF6673}>

Type HELP for debugger help, or (SB-EXT:EXIT) to exit from SBCL.

restarts (invokable by number or by possibly-abbreviated name):
  0: [RETRY                        ] Retry compiling
   #<PROTOBUF-SOURCE-FILE "opentelemetry-cl" "opentelemetry-proto" "resource">.
  1: [ACCEPT                       ] Continue, treating compiling
   #<PROTOBUF-SOURCE-FILE "opentelemetry-cl" "opentelemetry-proto" "resource">
                                     as having been successful.
  2:                                 Retry ASDF operation.
  3: [CLEAR-CONFIGURATION-AND-RETRY] Retry ASDF operation after resetting the
                                     configuration.
  4:                                 Retry ASDF operation.
  5:                                 Retry ASDF operation after resetting the
                                     configuration.
  6: [ABORT                        ] Give up on "opentelemetry-cl"
  7: [REGISTER-LOCAL-PROJECTS      ] Register local projects and try again.
  8: [CONTINUE                     ] Ignore runtime option --eval "(ros:run '((:eval\"(ros:quicklisp)\")(:system \"opentelemetry-cl\")))".
  9:                                 Skip rest of --eval and --load options.
 10:                                 Skip to toplevel READ/EVAL/PRINT loop.
 11: [EXIT                         ] Exit SBCL (calling #'EXIT, killing the process).

(CL-PROTOBUFS:PROTOBUF-ERROR "Could not find file ~S imported by ~S" "opentelemetry/proto/common/v1/common.proto" #<CL-PROTOBUFS:FILE-DESCRIPTOR RESOURCE (package opentelemetry.proto.resource.v1) {1004AF6673}>)
   source: (ERROR 'PROTOBUF-ERROR :FORMAT-CONTROL FORMAT-CONTROL
            :FORMAT-ARGUMENTS FORMAT-ARGUMENTS)

It's saying that it can't find the proto file for common.proto, but as far as I can tell, the search path is setup correctly. It's failing in the validate-imports function, but it's not exactly clear to me what I need to do to get common.proto to resolve in *file-descriptors*.

A few things I've noticed:

$ protoc \
  --proto_path=$PWD/opentelemetry-proto/ \
  --cl-pb_out=output-file=resource.lisp:$PWD \
  $PWD/opentelemetry-proto/opentelemetry/proto/resource/v1/resource.proto
$ ls -lah resource.lisp
-rw-r--r-- 1 mike 1.5K Feb 28 19:21 resource.lisp

Is there anything obvious that I may be doing wrong?

Thanks in advance.

Slids commented 6 months ago

I've never used qlot, so I cannot say. @cgay ?

cgay commented 6 months ago

No idea. Haven't used modern Lisp "in anger" outside of Google yet.

michaeldelago commented 6 months ago

qlot generally handles dependency management, but it's definitely worth a shot to test without it. I wrote this short script as load.lisp:

load.lisp ```lisp (ql:quickload :cl-protobufs) (ql:quickload :cl-protobufs.asdf) (asdf:load-asd (merge-pathnames (uiop:getcwd) "opentelemetry-cl.asd")) (ql:quickload :opentelemetry-cl) ```

and then ran it as sbcl --load load.lisp

Error output remains roughly the same:

Error Output ```console $ sbcl --load load.lisp To load "cl-protobufs": Load 1 ASDF system: cl-protobufs ; Loading "cl-protobufs" To load "cl-protobufs.asdf": Load 1 ASDF system: cl-protobufs.asdf ; Loading "cl-protobufs.asdf" To load "opentelemetry-cl": Load 1 ASDF system: opentelemetry-cl ; Loading "opentelemetry-cl" [package cl-protobufs.opentelemetry.proto.resource.v1] [package cl-protobufs.opentelemetry.proto.resource.v1]While evaluating the form starting at line 6, column 0 of #P"/home/mike/prog/opentelemetry-cl/load.lisp": debugger invoked on a CL-PROTOBUFS:PROTOBUF-ERROR in thread #: Could not find file "opentelemetry/proto/common/v1/common.proto" imported by # Type HELP for debugger help, or (SB-EXT:EXIT) to exit from SBCL. restarts (invokable by number or by possibly-abbreviated name): 0: [RETRY ] Retry compiling #. 1: [ACCEPT ] Continue, treating compiling # as having been successful. 2: Retry ASDF operation. 3: [CLEAR-CONFIGURATION-AND-RETRY] Retry ASDF operation after resetting the configuration. 4: Retry ASDF operation. 5: Retry ASDF operation after resetting the configuration. 6: [ABORT ] Give up on "opentelemetry-cl" 7: [REGISTER-LOCAL-PROJECTS ] Register local projects and try again. 8: [RETRY ] Retry EVAL of current toplevel form. 9: [CONTINUE ] Ignore error and continue loading file "/home/mike/prog/opentelemetry-cl/load.lisp". 10: Abort loading file "/home/mike/prog/opentelemetry-cl/load.lisp". 11: Ignore runtime option --load "load.lisp". 12: Skip rest of --eval and --load options. 13: Skip to toplevel READ/EVAL/PRINT loop. 14: [EXIT ] Exit SBCL (calling #'EXIT, killing the process). (CL-PROTOBUFS:PROTOBUF-ERROR "Could not find file ~S imported by ~S" "opentelemetry/proto/common/v1/common.proto" #) source: (ERROR 'PROTOBUF-ERROR :FORMAT-CONTROL FORMAT-CONTROL :FORMAT-ARGUMENTS FORMAT-ARGUMENTS) ```
michaeldelago commented 5 months ago

Okay, I've figured it out. It seems that the file descriptor for common.proto isn't being properly added to the file-descriptors hash table. In asdf.lisp, it looks like the filename is passed into protoc, meaning the generated lisp file contains (add-file-descriptor #P"common.proto" 'common). This doesn't properly get returned during the validate-imports function call for resource.proto, where the import is trying to import path relative to the root, opentelemetry/proto/common/v1/common.proto

At the moment, my workaround is to disable the check in the validate-imports function of define-proto.lisp:

176,177c176
<       (unless imported
<         (protobuf-error "Could not find file ~S imported by ~S" import file-descriptor)))))
---
>       )))

Not really sure how to concisely implement a meaningful unit test for this at the moment (past my bedtime :upside_down_face:), but that's where I've gotten thus far