structure for parser

[chavarera]

There are two options to store parsing level functions

Questions 1) inside utils with driver_parser.py

├── s_tool
│   ├── driver.py
│   └── utils
│       ├── driver_exceptions.py
│       ├── driver_parser.py
│       ├── driver_utils.py
│       ├── __init__.py

2) inside root directory with driver_parser.py

├── s_tool
│   ├── driver.py
│   ├── driver_parser.py
│   └── utils
│       ├── driver_exceptions.py
│       ├── driver_utils.py
│       ├── __init__.py

which one is best for optimization and future-use.

and we need to remove driver word from every utils files

[chavarera]

@aahnik whats your suggestion?

[aahnik]

and we need to remove driver word from every utils files

yup! I also thought that.

we can put the errors.py inside the package s_tool directly. (no need of nesting inside utils)

and we can have a utils.py

├── s_tool
│   ├── __init__.py
│   ├── errors.py # or exceptions.py
│   ├── driver.py
│   ├── parser.py
│   └── utils.py

Many popular packages have the following files directly in their package (no need for subpackage)

• logging.py or logger.py (to instantiate a custom logger)
• utils.py (for helper functions)
• errors.py or exceptions.py (for custom exceptions)
• typing.py or types.py (for custom type definitions)

By "popular" packages I mean fastapi, pydantic, and many many other

I propose that we remove the subpackage utils completely for now. Let's keep everything in the main super package. As the project matures more, we can take a decision later.

[aahnik]

If the code in parser.py goes above 200 lines, then we can create a sub-package parsers and create a module for each type of parser.

[chavarera]

If the code in parser.py goes above 200 lines, then we can create a sub-package parsers and create a module for each type of parser.

it will obviously goes above 200 line because parser will contain extraction logic for different type of HTML element

def select_options(element):
   pass

def extract_table(element):
   pass

def form_element(element):
   pass

def hidden_element(element):
   pass

def input_boxes(element):
   pass

should we create separate functions or map with a Parser class

[chavarera]

and we need to remove driver word from every utils files

yup! I also thought that.

we can put the errors.py inside the package s_tool directly. (no need of nesting inside utils)

and we can have a utils.py

├── s_tool
│   ├── __init__.py
│   ├── errors.py # or exceptions.py
│   ├── driver.py
│   ├── parser.py
│   └── utils.py

Many popular packages have the following files directly in their package (no need for subpackage)

* logging.py or logger.py (to instantiate a custom logger)

* utils.py (for helper functions)

* errors.py or exceptions.py (for custom exceptions)

* typing.py or types.py (for custom type definitions)

By "popular" packages I mean fastapi, pydantic, and many many other

I propose that we remove the subpackage utils completely for now. Let's keep everything in the main super package. As the project matures more, we can take a decision later.

I am changing project structure according to this suggestion