search

HASTE-project / hom-2

HASTE-o-matic: Interactive, Stream-based Analysis of Large Datasets with Information Hierarchies
1 stars 1 forks source link

Manager should persist state #21

Open benblamey opened 2 years ago

benblamey commented 2 years ago

K8 should manage persistence of its own resources.

Suggested implementation is on startup to retrieve list of pods and deployments, which have GUIDs as names. A key-value store can be used to persist state of each corresponding Tier instance in the manager.

So on startup, if any tiers exist, the state can be re-created.

Haoyuan-L commented 2 years ago

Hi, can you share the Dockerfile of the manager? I need to update the image and deploy some manager-related solutions

Haoyuan-L commented 2 years ago

hom-impl-2.manager

benblamey commented 2 years ago

Sure, it is

https://github.com/HASTE-project/hom-2/blob/2b9a5d64a7c2f8049aaf9ed7da98d5fd5e850365/manager/Dockerfile

On Mon, 11 Jul 2022, 17:13 Haoyuan Li, @.***> wrote:

hom-impl-2.manager

— Reply to this email directly, view it on GitHub https://github.com/HASTE-project/hom-2/issues/21#issuecomment-1180537269, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAF7BSYE4UCRU3LKMB3NX3TVTQ223ANCNFSM5XJV32BA . You are receiving this because you authored the thread.Message ID: @.***>

Haoyuan-L commented 2 years ago

Sure, it is https://github.com/HASTE-project/hom-2/blob/2b9a5d64a7c2f8049aaf9ed7da98d5fd5e850365/manager/Dockerfile … On Mon, 11 Jul 2022, 17:13 Haoyuan Li, @.> wrote: hom-impl-2.manager — Reply to this email directly, view it on GitHub <#21 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAF7BSYE4UCRU3LKMB3NX3TVTQ223ANCNFSM5XJV32BA . You are receiving this because you authored the thread.Message ID: @.>

Thanks! About the JAR file (build/libs/hom-impl-2.manager-1.0-SNAPSHOT.jar), is this just package all the java files in the repository: http://hom-2/manager/src/main/java/com/benblamey/hom/manager/?

benblamey commented 2 years ago

The idea is that the list of tiers inside the manager, should be externally persisted so that the manager can be restarted without loss of state.

There are various frameworks to assist with persistence of Java objects

On Tue, 12 Jul 2022, 10:41 Ben Blamey, @.***> wrote:

Yea it's built with gradle. Install intelliJ and it should work.

On Mon, 11 Jul 2022, 20:50 Haoyuan Li, @.***> wrote:

Sure, it is https://github.com/HASTE-project/hom-2/blob/2b9a5d64a7c2f8049aaf9ed7da98d5fd5e850365/manager/Dockerfile … <#m_3151139181903547474m-2880410000165966211_> On Mon, 11 Jul 2022, 17:13 Haoyuan Li, @.> wrote: hom-impl-2.manager — Reply to this email directly, view it on GitHub <#21 (comment) https://github.com/HASTE-project/hom-2/issues/21#issuecomment-1180537269>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAF7BSYE4UCRU3LKMB3NX3TVTQ223ANCNFSM5XJV32BA https://github.com/notifications/unsubscribe-auth/AAF7BSYE4UCRU3LKMB3NX3TVTQ223ANCNFSM5XJV32BA . You are receiving this because you authored the thread.Message ID: @.>

Thanks! About the JAR file ( build/libs/hom-impl-2.manager-1.0-SNAPSHOT.jar), is this just package all the java files in the repository: http://hom-2/manager/src/main/java/com/benblamey/hom/manager/?

— Reply to this email directly, view it on GitHub https://github.com/HASTE-project/hom-2/issues/21#issuecomment-1180749284, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAF7BS45T5GVGFE6JYVFQT3VTRUHJANCNFSM5XJV32BA . You are receiving this because you authored the thread.Message ID: @.***>

benblamey commented 2 years ago

Place to start would be to implement XML serialization on tier and it's subclasses

On Tue, 12 Jul 2022, 10:48 Ben Blamey, @.***> wrote:

The idea is that the list of tiers inside the manager, should be externally persisted so that the manager can be restarted without loss of state.

There are various frameworks to assist with persistence of Java objects

On Tue, 12 Jul 2022, 10:41 Ben Blamey, @.***> wrote:

Yea it's built with gradle. Install intelliJ and it should work.

On Mon, 11 Jul 2022, 20:50 Haoyuan Li, @.***> wrote:

Sure, it is https://github.com/HASTE-project/hom-2/blob/2b9a5d64a7c2f8049aaf9ed7da98d5fd5e850365/manager/Dockerfile … <#m_2658545846834994555_m_3151139181903547474m-2880410000165966211_> On Mon, 11 Jul 2022, 17:13 Haoyuan Li, @.> wrote: hom-impl-2.manager — Reply to this email directly, view it on GitHub <#21 (comment) https://github.com/HASTE-project/hom-2/issues/21#issuecomment-1180537269>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAF7BSYE4UCRU3LKMB3NX3TVTQ223ANCNFSM5XJV32BA https://github.com/notifications/unsubscribe-auth/AAF7BSYE4UCRU3LKMB3NX3TVTQ223ANCNFSM5XJV32BA . You are receiving this because you authored the thread.Message ID: @.

Thanks! About the JAR file ( build/libs/hom-impl-2.manager-1.0-SNAPSHOT.jar), is this just package all the java files in the repository: http://hom-2/manager/src/main/java/com/benblamey/hom/manager/?

— Reply to this email directly, view it on GitHub https://github.com/HASTE-project/hom-2/issues/21#issuecomment-1180749284, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAF7BS45T5GVGFE6JYVFQT3VTRUHJANCNFSM5XJV32BA . You are receiving this because you authored the thread.Message ID: @.***>

benblamey commented 2 years ago

Maybe try Jackson for example

https://www.baeldung.com/jackson-xml-serialization-and-deserialization

On Tue, 12 Jul 2022, 11:52 Ben Blamey, @.***> wrote:

Place to start would be to implement XML serialization on tier and it's subclasses

On Tue, 12 Jul 2022, 10:48 Ben Blamey, @.***> wrote:

The idea is that the list of tiers inside the manager, should be externally persisted so that the manager can be restarted without loss of state.

There are various frameworks to assist with persistence of Java objects

On Tue, 12 Jul 2022, 10:41 Ben Blamey, @.***> wrote:

Yea it's built with gradle. Install intelliJ and it should work.

On Mon, 11 Jul 2022, 20:50 Haoyuan Li, @.***> wrote:

Sure, it is https://github.com/HASTE-project/hom-2/blob/2b9a5d64a7c2f8049aaf9ed7da98d5fd5e850365/manager/Dockerfile … <#m_-6197363225623025062_m_2658545846834994555_m_3151139181903547474m-2880410000165966211_> On Mon, 11 Jul 2022, 17:13 Haoyuan Li, @.> wrote: hom-impl-2.manager — Reply to this email directly, view it on GitHub <#21 (comment) https://github.com/HASTE-project/hom-2/issues/21#issuecomment-1180537269>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAF7BSYE4UCRU3LKMB3NX3TVTQ223ANCNFSM5XJV32BA https://github.com/notifications/unsubscribe-auth/AAF7BSYE4UCRU3LKMB3NX3TVTQ223ANCNFSM5XJV32BA . You are receiving this because you authored the thread.Message ID: @.

Thanks! About the JAR file ( build/libs/hom-impl-2.manager-1.0-SNAPSHOT.jar), is this just package all the java files in the repository: http://hom-2/manager/src/main/java/com/benblamey/hom/manager/?

— Reply to this email directly, view it on GitHub https://github.com/HASTE-project/hom-2/issues/21#issuecomment-1180749284, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAF7BS45T5GVGFE6JYVFQT3VTRUHJANCNFSM5XJV32BA . You are receiving this because you authored the thread.Message ID: @.***>

Haoyuan-L commented 2 years ago

Thanks! I will try

Haoyuan-L commented 2 years ago

Hi Ben, I tried to serialize tier and its subclasses to XML files (in manager.java when adding tiers). My current solution for serialization is to write the tier configuration externally in an XML file when the manager adds tiers. Every time I restart the manager, the tier state can be rebuilt with the XML file if exist. Am I right? If so, how can I test my code? (to check the XML file locally)

benblamey commented 2 years ago

Hey Haoyuan! Awesome! Yes that sounds like a great solution. Also when tiers are removed?

For testing, a unit test can load am XML file as a jar, and deserialize it. Assert the number of tiers and their properties. And then backwards.

Did you push a branch?

On Wed, 20 Jul 2022, 18:54 Haoyuan Li, @.***> wrote:

Hi Ben, I tried to serialize tier and its subclasses to XML files (in manager.java when adding tiers). My current solution for serialization is to write the tier configuration externally in an XML file when the manager adds tiers. Every time I restart the manager, the tier state can be rebuilt with the XML file if exist. Am I right? If so, how can I test my code? (to check the XML file locally)

— Reply to this email directly, view it on GitHub https://github.com/HASTE-project/hom-2/issues/21#issuecomment-1190522315, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAF7BS2Z2PQXJLBLHGZ7IZ3VVAVMJANCNFSM5XJV32BA . You are receiving this because you authored the thread.Message ID: @.***>

Haoyuan-L commented 2 years ago

Not yet, I'm still working on the code. I will push a branch after I test it locally. My initial thought is to remove the tier configurations from the XML file (based on the uniqueID) when the user removes the specific tier, but I haven't deployed it.

benblamey commented 2 years ago

Cool!

It might be cleaner just to re-serialize the whole list on each change, and overwrote the file. The cost is trivial. It's much cleaner that way.

On Wed, 20 Jul 2022, 19:47 Haoyuan Li, @.***> wrote:

Not yet, I'm still working on the code. I will push a branch after I test it locally. My initial thought is to remove the tier configurations from the XML file (based on the uniqueID) when the user removes the specific tier, but I

— Reply to this email directly, view it on GitHub https://github.com/HASTE-project/hom-2/issues/21#issuecomment-1190575110, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAF7BS32Q72ITEMDGQ3ISYDVVA3RLANCNFSM5XJV32BA . You are receiving this because you authored the thread.Message ID: @.***>

Haoyuan-L commented 2 years ago

Cool! It might be cleaner just to re-serialize the whole list on each change, and overwrote the file. The cost is trivial. It's much cleaner that way. … On Wed, 20 Jul 2022, 19:47 Haoyuan Li, @.> wrote: Not yet, I'm still working on the code. I will push a branch after I test it locally. My initial thought is to remove the tier configurations from the XML file (based on the uniqueID) when the user removes the specific tier, but I — Reply to this email directly, view it on GitHub <#21 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAF7BS32Q72ITEMDGQ3ISYDVVA3RLANCNFSM5XJV32BA . You are receiving this because you authored the thread.Message ID: @.>

yea, it seems more efficient, thanks!

Haoyuan-L commented 2 years ago

Hi Ben, can you check the latest pull request #32 ? I have tested it successfully, solved issue #21 and issue #18

benblamey commented 2 years ago

Hi Haoyuan,

I have merged some of your changes into the main repo. Maybe do a fetch and check what's updated.

On Fri, 29 Jul 2022, 20:27 Haoyuan Li, @.***> wrote:

Hi Ben, can you check the latest pull request #32 https://github.com/HASTE-project/hom-2/pull/32 ? I have tested it successfully, solved issue #21 https://github.com/HASTE-project/hom-2/issues/21 and issue #18 https://github.com/HASTE-project/hom-2/issues/18

— Reply to this email directly, view it on GitHub https://github.com/HASTE-project/hom-2/issues/21#issuecomment-1199833614, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAF7BS4O5APVIYFTLRDCMKDVWQO7NANCNFSM5XJV32BA . You are receiving this because you authored the thread.Message ID: @.***>

benblamey commented 2 years ago

Yea it's built with gradle. Install intelliJ and it should work.

On Mon, 11 Jul 2022, 20:50 Haoyuan Li, @.***> wrote:

Sure, it is https://github.com/HASTE-project/hom-2/blob/2b9a5d64a7c2f8049aaf9ed7da98d5fd5e850365/manager/Dockerfile … <#m-2880410000165966211> On Mon, 11 Jul 2022, 17:13 Haoyuan Li, @.> wrote: hom-impl-2.manager — Reply to this email directly, view it on GitHub <#21 (comment) https://github.com/HASTE-project/hom-2/issues/21#issuecomment-1180537269>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAF7BSYE4UCRU3LKMB3NX3TVTQ223ANCNFSM5XJV32BA https://github.com/notifications/unsubscribe-auth/AAF7BSYE4UCRU3LKMB3NX3TVTQ223ANCNFSM5XJV32BA . You are receiving this because you authored the thread.Message ID: @.>

Thanks! About the JAR file (build/libs/hom-impl-2.manager-1.0-SNAPSHOT.jar), is this just package all the java files in the repository: http://hom-2/manager/src/main/java/com/benblamey/hom/manager/?

— Reply to this email directly, view it on GitHub https://github.com/HASTE-project/hom-2/issues/21#issuecomment-1180749284, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAF7BS45T5GVGFE6JYVFQT3VTRUHJANCNFSM5XJV32BA . You are receiving this because you authored the thread.Message ID: @.***>