search

determined-ai / determined

Determined is an open-source machine learning platform that simplifies distributed training, hyperparameter tuning, experiment tracking, and resource management. Works with PyTorch and TensorFlow.
https://determined.ai
Apache License 2.0
3.01k stars 350 forks source link

make -C proto fails on Fedora 36 #4483

Closed sirredbeard closed 2 years ago

sirredbeard commented 2 years ago

Summary

make -C proto fails on Fedora 36

Environment

Fedora 36 node v16.14.0 protoc 3.19.4 python 3.17.13 go 1.18.3

Steps to Reproduce

Install Fedora 36 Install node, protobuf-compiler, python3.7, and go via Fedora package repository. Clone Determined repository: git clone --recurse-submodules https://github.com/determined-ai/determined.git Enter directory: cd determined Create venv: python3.7 -m venv ~/.virtualenvs/determined Activate: . ~/.virtualenvs/determined/bin/activate Set PATH export PATH=$PATH:$HOME/go/bin Run make: make all

Expected Results

Determined builds.

Actual Results

The protobuf generation script fails:

make -C proto build
make[2]: Entering directory '/home/hayden/determined/proto'
rm -rf build/proto pkg
mkdir -p build/proto
# Protobuf generation.
for source in src/determined/resourcepool/v1/resourcepool.proto src/determined/model/v1/model.proto src/determined/workspace/v1/workspace.proto src/determined/job/v1/job.proto src/determined/util/v1/util.proto src/determined/device/v1/device.proto src/determined/user/v1/user.proto src/determined/checkpoint/v1/checkpoint.proto src/determined/command/v1/command.proto src/determined/master/v1/master.proto src/determined/container/v1/container.proto src/determined/tensorboard/v1/tensorboard.proto src/determined/api/v1/agent.proto src/determined/api/v1/user.proto src/determined/api/v1/api.proto src/determined/api/v1/command.proto src/determined/api/v1/auth.proto src/determined/api/v1/pagination.proto src/determined/api/v1/resourcepool.proto src/determined/api/v1/trial.proto src/determined/api/v1/master.proto src/determined/api/v1/model.proto src/determined/api/v1/workspace.proto src/determined/api/v1/tensorboard.proto src/determined/api/v1/shell.proto src/determined/api/v1/project.proto src/determined/api/v1/template.proto src/determined/api/v1/task.proto src/determined/api/v1/checkpoint.proto src/determined/api/v1/job.proto src/determined/api/v1/notebook.proto src/determined/api/v1/experiment.proto src/determined/template/v1/template.proto src/determined/agent/v1/agent.proto src/determined/task/v1/task.proto src/determined/experiment/v1/searcher.proto src/determined/experiment/v1/experiment.proto src/determined/trial/v1/trial.proto src/determined/notebook/v1/notebook.proto src/determined/project/v1/project.proto src/determined/log/v1/log.proto src/determined/shell/v1/shell.proto ; do protoc -I src "$source" --go_out=plugins=grpc:build/proto ; done
google/protobuf/descriptor.proto: File not found.
google/protobuf/any.proto: File not found.
protoc-gen-swagger/options/openapiv2.proto:7:1: Import "google/protobuf/any.proto" was not found or had errors.
protoc-gen-swagger/options/openapiv2.proto:234:5: "google.protobuf.Any" is not defined.
protoc-gen-swagger/options/annotations.proto:7:1: Import "google/protobuf/descriptor.proto" was not found or had errors.
protoc-gen-swagger/options/annotations.proto:8:1: Import "protoc-gen-swagger/options/openapiv2.proto" was not found or had errors.
protoc-gen-swagger/options/annotations.proto:10:8: "google.protobuf.FileOptions" is not defined.

Complete build script is here.

ioga commented 2 years ago

does fedora happen to have an analogue of libprotobuf-dev, as suggested here?

sirredbeard commented 2 years ago

Searching for protobuf, I can see a golang-google-protobuf and golang-github-protobuf-devel but installing them doesn't seem to make a difference.

ioga commented 2 years ago

The missing file google/protobuf/any.proto is not golang-specific but sort of a "standard library" for protobufs regardless of the language. Maybe protobuf-devel? Any way to search fedora package manager by filename / file path?

Also you could check if you already have that file installed in your system, in which case the question will become why protoc doesn't see it.

sirredbeard commented 2 years ago

Odd, golang-github-protobuf-devel provides any.go, along with duration.go, timestamp.go, etc. but not *.proto`.

golang-github-gogo-protobuf-devel, a fork of protobuf, does provide any.proto.

sirredbeard commented 2 years ago

Closing in favor of another solution.