I have seen a lot of research that does not have a corresponding open source project, and are hard for anyone to follow up on the results. But this is completely open. And as far as I am concerned it's great work.
The nagging question is, why don't I see more adoption. The biggest contribution being the algebra and the transformations it makes available.
This algebra could potentially be included in many programming languages as part of the optimization engine. The problems of Big data, I believe are analogous to multiprocessor environments, where data may not necessarily be broadcasted. I don't think this is a specialty tool. I want to use it in general purpose IT systems programming. Possibly for declarative low code programming. Where the code would otherwise not be very performant.
From my perspective separating optimization as much as possible would be very positive. In my mind, optimization is extremely case driven. It would be ideal to have a generic interface that would allow tool makers to create and modify existing cost functions. Even possibly doing global optimization. Separating algebra, optimization algorithm and cost functions. Through a clear and concise interface would be a huge win.
As an example the <- to <-- operator. Defining small data vs big data set. It's not part of the algebra, but I think it could be specified as an attribute of the collection itself. As such the cost function would need to be aware of it. But not necessarily the optimizer.
Because I "live" in the Microsoft dot net world. I could see migrating the algebra engine almost verbatim to F#. My focus is the optimizer and C# language interface. At first I would use a Breadth-first search algorithm as the optimizer. And try all the transformations possible. I'll try to have a simple cost function. And I'll try to create documentation for the average tool maker.
I hope this all sounds well to you. This will be done as a side project for me, and I very much regard your experience in this field. Any insights you could give me will be greatly appreciated. When I have some work done I will post something here as well.
I love your work. There are so many pluses.
I have seen a lot of research that does not have a corresponding open source project, and are hard for anyone to follow up on the results. But this is completely open. And as far as I am concerned it's great work.
The nagging question is, why don't I see more adoption. The biggest contribution being the algebra and the transformations it makes available.
This algebra could potentially be included in many programming languages as part of the optimization engine. The problems of Big data, I believe are analogous to multiprocessor environments, where data may not necessarily be broadcasted. I don't think this is a specialty tool. I want to use it in general purpose IT systems programming. Possibly for declarative low code programming. Where the code would otherwise not be very performant.
From my perspective separating optimization as much as possible would be very positive. In my mind, optimization is extremely case driven. It would be ideal to have a generic interface that would allow tool makers to create and modify existing cost functions. Even possibly doing global optimization. Separating algebra, optimization algorithm and cost functions. Through a clear and concise interface would be a huge win.
As an example the <- to <-- operator. Defining small data vs big data set. It's not part of the algebra, but I think it could be specified as an attribute of the collection itself. As such the cost function would need to be aware of it. But not necessarily the optimizer.
Because I "live" in the Microsoft dot net world. I could see migrating the algebra engine almost verbatim to F#. My focus is the optimizer and C# language interface. At first I would use a Breadth-first search algorithm as the optimizer. And try all the transformations possible. I'll try to have a simple cost function. And I'll try to create documentation for the average tool maker.
I hope this all sounds well to you. This will be done as a side project for me, and I very much regard your experience in this field. Any insights you could give me will be greatly appreciated. When I have some work done I will post something here as well.
Thank you for your contribution!!!