search

googleapis / gapic-generator-python

Generate Python API client libraries from Protocol Buffers.
Apache License 2.0
121 stars 66 forks source link

gapic-generator-python fails to complete with a sufficiently large protobuf tree #950

Closed tmc closed 2 months ago

tmc commented 3 years ago

This generator fails when there is a type reference to a very large protobuf type tree. (For example, the types described at https://github.com/google/fhir/tree/master/proto/google/fhir/proto ).

Initial debugging appears to show that MessageType.with_context is where the majority of the time is spent.

software-dov commented 3 years ago

MessageType.with_context is also a colossal memory hog, which may be the actual cause. I have definitely generated OOMs when attempting to compile particular APIs. This re-prioritizes an undertaking to do less copying that I'm just about to restart.

tmc commented 2 years ago

@software-dov Curious if you have anything to share regarding this.

tmc commented 2 years ago

@software-dov I experimented and by moving away from FrozenSet and by ensuring the same set() instance is threaded through the service/message/field tree the generation time+memory goes down from non-terminating and consuming 32G to finishing within 2 minutes.

vam-google commented 2 years ago

Waiting for https://github.com/googleapis/gapic-generator-python/pull/1125 to be merged. @tmc can you please follow up on it? Thanks!

tmc commented 2 years ago

@vam-google I think that PR needs review.

vchudnov-g commented 2 months ago

It seems like this was fixed by #1125 . If not, please file a new issue so we can follow up. Thanks!