search

airavata-courses / neo

A microservice architecture based weather feed visualization platform.
http://projectneo.ignorelist.com:32000/
Apache License 2.0
1 stars 1 forks source link

Feature - Develop Data Processing Service #19

Closed rajdeepc2792 closed 2 years ago

rajdeepc2792 commented 2 years ago

As planned, this service will be fetching data according to the selected weather attributes(such as radar-location, date-time and other attributes like reflectivity) from data ingestion service. Process the data and provide back the resulted data.

niravraje commented 2 years ago

Hey @shanki07, Can you share the attributes (or even the data types of those attributes) you are using from the S3 fetched data? For example, reflectivity, etc. If you already have a list of all attributes existing, that would be even better. I have got a simple gRPC communication working between 2 Python microservices and wanted to prepare the Protocol Buffer files (basically need to define the schema) for communication between gateway and this service.

shanki07 commented 2 years ago

Hi @niravraje ,

Please find the list of attributes that we can add a parameter field in the UI, this contains data type, their units, and all kind of metadata related to that field that is present in data, but we will mostly just transfer the numpy array as JSON serialized data:

fields: reflectivity: data: <ndarray of type: float32 and shape: (3960, 1832)> units: dBZ standard_name: equivalent_reflectivity_factor long_name: Reflectivity valid_max: 94.5 valid_min: -32.0 coordinates: elevation azimuth range _FillValue: -9999.0 velocity: data: <ndarray of type: float32 and shape: (3960, 1832)> units: meters_per_second standard_name: radial_velocity_of_scatterers_away_from_instrument long_name: Mean doppler Velocity valid_max: 95.0 valid_min: -95.0 coordinates: elevation azimuth range _FillValue: -9999.0 spectrum_width: data: <ndarray of type: float32 and shape: (3960, 1832)> units: meters_per_second standard_name: doppler_spectrum_width long_name: Spectrum Width valid_max: 63.0 valid_min: -63.5 coordinates: elevation azimuth range _FillValue: -9999.0 differential_reflectivity: data: <ndarray of type: float32 and shape: (3960, 1832)> units: dB standard_name: log_differential_reflectivity_hv long_name: log_differential_reflectivity_hv valid_max: 7.9375 valid_min: -7.875 coordinates: elevation azimuth range _FillValue: -9999.0 cross_correlation_ratio: data: <ndarray of type: float32 and shape: (3960, 1832)> units: ratio standard_name: cross_correlation_ratio_hv long_name: Cross correlation_ratio (RHOHV) valid_max: 1.0 valid_min: 0.0 coordinates: elevation azimuth range _FillValue: -9999.0 differential_phase: data: <ndarray of type: float32 and shape: (3960, 1832)> units: degrees standard_name: differential_phase_hv long_name: differential_phase_hv valid_max: 360.0 valid_min: 0.0 coordinates: elevation azimuth range _FillValue: -9999.0

I think we are currently we can target Reflectivity, Velocity, and spectrum_width.

Please let me know if this doesn't make sense to you we can discuss this in more detail.

Regards, Shashank

niravraje commented 2 years ago

Thanks @shanki07, this is helpful. @rajdeepc2792 - We discussed on keeping one microservice for data processing and S3 ingestion to avoid over-complicating the architecture right now with message-queues between these two. They also seem to be highly dependent on each other right now. We could decouple them later in case our processing microservice has to perform any heavy computation or ML-related work in future (which seems unlikely). What do you think?

shanki07 commented 2 years ago

As we have found out, transferring the data over microservices would not be the best approach as the file size is around 50MBs when converted to JSON and 30 MBs when converted to a binary file. We can achieve the size of 0.5MB when we use the numpy compression method but sharing visualization would be a much faster and more efficient way as of now.

niravraje commented 2 years ago

Hi @shanki07, I have created a data_viz_engine.py file for your data processing/ingestion code and used app.py for the writing the Servicer for grpc communication with the gateway. Can you please review and confirm if everything looks good for your code? If all looks good, you can pull these changes for future enhancements to the data processing scripts

shanki07 commented 2 years ago

Hi @niravraje,

It looks good to me, Sure I will fetch the changes that you have made in my local copy as well . Thanks!