GSoC 2022 Idea: Mapping existing configuration parameters to the APIs

harikrishna-patnala commented 3 years ago

Background:

Cloudstack has global settings in which there are around 672 configuration parameters. Using these parameters one can adjust the values according to the need of the environment. For example "allow.duplicate.networkname" default value of this parameter is true, which means networks can be created with same name in an account. If the value is set to false then duplicate network names are not allowed.

Problem statement: When admin wants to change the API behaviour or during debugging an issue, admin has to search for the corresponding configuration parameter. The only way currently everyone is using is by using string search or looking into corresponding documentation. Searching over 672 configuration parameters which is a huge list is not straight forward or may lead to missing few parameters.

Solution: To address this problem I would like to propose the solution which maps the configurations parameters to the corresponding APIs. One can know what are the configuration parameters involved for a specific API. For example "createNetwork" API will be mapped to "allow.duplicate.networkname". When admin wants to see what are the configuration parameters used for "createNetwork" API , this mapping will help. In the final result we will have a table with APIs in one column and configuration parameters in another column.

Proposed Tasks

Get started on basic CloudStack codebase and development (building and running CloudStack)
Understand the marvin test framework on how to setup and run tests
Setup any hypervisor (VMware/KVM/XenServer) based CloudStack dev/test environment
Understand how global configurations are defined and used in CloudStack
Use marvin tests to call all the APIs and record the configuration parameters used for each API call by using wrapper around configuration framework
Deliverables: documentation and community pull request, end-to-end demo of feature

Relevant Skills

Java and Python
Virtualization/Cloud domain knowledge and usage

Proposed Mentor

Harikrishna Patnala - Committer Apache CloudStack

Difficulty

Major

Potential Mentors

Harikrishna Patnala - Committer Apache CloudStack Any developer from the community

Example and references

https://cwiki.apache.org/confluence/display/CLOUDSTACK/Configuration https://cwiki.apache.org/confluence/display/CLOUDSTACK/Marvin+-+Testing+with+Python#MarvinTestingwithPython-Installation

ISSUE TYPE

Feature Idea

damanBirSingh commented 3 years ago

Hello, I am an ME student from Bits Pilani, Goa, India. I would love to contribute to an open source project like Cloudstack. I am currently going through the Cloudstack/Hackerbook and would start to draft a proposal asap. Furthermore, I also got a reply from @harikrishna-patnala with some resources to follow and will definitely go through those. Please let me know if there is anything else I should know.

Thanks & Regards, Daman

harikrishna-patnala commented 3 years ago

Hi @damanBirSingh, It's great to hear from you and your interest on this project. I'm happy to mentor and work with you. Like we already discussed please go through the links provided and start drafting the application as the end date is reaching soon.

You might have already done the following, if not please do it. I would advise that you read the community GSoC guidelines (https://community.apache.org/gsoc.html), join the dev mailing list (http://cloudstack.apache.org/mailing-lists.html) to discuss your GSoC idea/proposal/application, and start drafting your application (for example https://google.github.io/gsocguides/student/proposal-example-1). With your proposed mentor, start an early discussion with them and you can ask them to review your application draft (for example, compose your draft and share with a Google doc).

I can help you with drafting application. Please let me know. Thanks

khamarshaikh commented 2 years ago

@harikrishna-patnala , is this feature still open ? If yes , i would like to contribute to it.

rohityadavcloud commented 2 years ago

cc @sureshanaparti you have a global settings re-mapping/categorisation PR, is this related to your PR, or this issue isn't needed anymore as it may be complex for a student project @harikrishna-patnala ?

sureshanaparti commented 2 years ago

cc @sureshanaparti you have a global settings re-mapping/categorisation PR, is this related to your PR, or this issue isn't needed anymore as it may be complex for a student project @harikrishna-patnala ?

@rohityadavcloud this is not related to global settings categorisation.

harikrishna-patnala commented 2 years ago

@rohityadavcloud, this is not related to @sureshanaparti 's work and I think this is not a complex one as it requires understanding only how to run marvin tests and a small wrapper around configuration framework.

Damans227 commented 2 years ago

@harikrishna-patnala Ok, I read through the ConfigKey<T> Class from cloud-framework-config package and understood how we can use it to declare a configuration parameter in an interface and grab its value using value() method. I have some questions to understand the workflow of the key management further:

1 Is cloudstack/api the package which comprises all the apis where we want to expose the global configuration parameters? 2 You have mentioned in the solution above that "In the final result we will have a table with APIs in one column and configuration parameters in another column" So is this going to be a new table we will create in the database or is it the existing configuration table we want to store this information in?

Thanks!

harikrishna-patnala commented 2 years ago

Hi @Damans227, great to see your interest here and the learning you have gone through. To answer your questions

Yes, most of the APIs are here, but there are other places as well like in plugins/
Either ways we can do, with the new table or with new columns in existing "configurations" table.

harikrishna-patnala commented 2 years ago

Hi @khamarshaikh, are you still looking to work on this ?

khamarshaikh commented 2 years ago

@harikrishna-patnala , Yes I am looking forward to work on this. I would be proposing a design document for the changes proposed. Is it ok if I can share it here or any specific channel ? Before finalising on implementation

Damans227 commented 2 years ago

Hi @Damans227, great to see your interest here and the learning you have gone through. To answer your questions

Yes, most of the APIs are here, but there are other places as well like in plugins/

Either ways we can do, with the new table or with new columns in existing "configurations" table.

Thanks for replying.

borisstoyanov commented 2 years ago

@harikrishna-patnala can you please indicate if this is medium or large project (175/350hrs)

harikrishna-patnala commented 2 years ago

@borisstoyanov to me it looks like Large project, because students have to understand or do the following (keeping these sections to let students go through this)

Know how to run smoke/integration tests
Work on idea to link API operation and config key
Write a wrapper around configDepot
Save values to DB

rohityadavcloud commented 2 years ago

This would be very complex for a student, it's going to require more hard-work than really opportunity to learn/do something interesting for students. Furthermore, I think @sureshanaparti 's PR/feature that groups global settings will make working with global settings easier in future - I think this can parked for future work @harikrishna-patnala probably not suitable/fun for students.

harikrishna-patnala commented 2 years ago

Okay @rohityadavcloud makes sense. I'll take it out from GSoC.

rohityadavcloud commented 2 years ago

Closing this, we have a PR to streamline global settings access in API/UI https://github.com/apache/cloudstack/pull/5797

apache / cloudstack