weekly | JJ

[vamseeachanta]

Objective Track the progress summary in a single issue board.

Example Format: WE 2024-04-26:

• API input data -> detailed info.

[JayachandraJangiti]

@vamseeachanta

Good morning Sir,

I have mailed you the extracting code (with virtual machine).

UML diagram of our work is completed using plantuml

You can find it below : https://github.com/JayachandraJangiti/WEB_SCRAPPING/blob/Extracting_table/Extract_table_content.plantuml

[JayachandraJangiti]

@vamseeachanta

Good morning Sir,

Here is the code :

https://github.com/JayachandraJangiti/WEB_SCRAPPING/blob/Extracting_table/Extract_table_content.py

Here is the setup file :

https://github.com/JayachandraJangiti/WEB_SCRAPPING/blob/Extracting_table/requirements.txt

You can run the Extract_table_content code using this setup file (contains all packages used with their versions).

[JayachandraJangiti]

@vamseeachanta

Good morning Sir,

I have changed code to take all inputs from configuration (.yml) file. It takes all inputs from the file given and run the program for each input.

Here is the code :

https://github.com/JayachandraJangiti/WEB_SCRAPPING/blob/Extracting_table/main.py

Here is the Output :

1) First of all It asks for the path of inputs file (.yml file). We need to give the path of .yml file in which inputs were written .

I have given below file as input

https://github.com/JayachandraJangiti/WEB_SCRAPPING/blob/Extracting_table/Inputs.yml

The inputs are

2) Then It prints dataframes for each input you have given in input file.

[vamseeachanta]

@JayachandraJangiti , I have rewired the data into the repository. Any future work should happen here.

Please see setup instructions below and make sure you can run the test file. https://github.com/vamseeachanta/energydata/blob/bseedata/dev_tools/dev_instructions.md

Feel free to contact @saiachanta2728 or @samdansk2 if you need help in this regard.

[vamseeachanta]

short-term tasks:

• run in energydata repo. In branch bseedata
• get more data (well properties etc.)
• Review plantuml drawing and bring to appropriate repo.

long-term tasks: a/ Run selenium code in background (without opening browser in foreground) b/ rewire selenium code to get data in a generic way. transfer code to assetutilities c/ function inside a class function is confusing and not generic. Rewire to generic utilities. d/ can we grab csv directly from UI by displaying there instead of downloading? Dependent on the website architecture.

[vamseeachanta]

@JayachandraJangiti , please give weekly updates. Else, accounting for work (and payments) will be difficult.

Also, did you get the energydata repo going? Did you get the virtual environment going? Need more prompt communications from you.

[vamseeachanta]

self-installing library instructions are given in this link: https://github.com/vamseeachanta/py_package

[JayachandraJangiti]

@vamseeachanta @saiachanta2728

Good morning Sir,

I and Sai have discussed, solved some problems and have done the following :

1) Created and worked with virtual Environments. 2) Had some problems like(Setting Environment variables in the laptop etc.) with miniconda3 and resolved those. 3) Cloned and Worked on both assetutilities and energydata repos. 4) Have gone to run .yml files, but we got stuck there (We got some errors).

[vamseeachanta]

@JayachandraJangiti @saiachanta2728 , share screenshot of the errors if you can and I can suggest tips. Else, we will have to do this in another meeting.