eco-bone / dataset-automation

Software to create any text based dataset using openAI api
1 stars 0 forks source link

Finish the automation project for my particular use case. #13

Open eco-bone opened 1 year ago

eco-bone commented 1 year ago

Finish creating an actual working software for my use case.

This will help me ensure that what I am doing will actually work and what issues I might face in the api interaction process.

Generalisation to be done under another issue after this is completed. Might even create new ms with generalisation from the start instead of trying to update the current ms.

eco-bone commented 1 year ago

The following is the output from chatgpt.

Need to extract content from it.

{ "id": "chatcmpl-8Mw6567T84JAEhxOGes8eyVsZ8Ew4", "object": "chat.completion", "created": 1700477233, "model": "gpt-3.5-turbo-0613", "choices": [ { "index": 0, "message": { "role": "assistant", "content": "1. Communication Skills: Effective communication is crucial for teachers to convey information clearly and engage with students, parents, and colleagues. This includes the ability to listen actively, explain complex concepts in a simple manner, and adapt communication style to various audiences.\n\n2. Classroom Management: Teachers need to establish a positive learning environment and maintain discipline. Strong classroom management skills involve setting clear expectations, enforcing rules consistently, and addressing disruptions promptly and fairly.\n\n3. Subject Knowledge: Proficiency in the subject matter being taught is essential for teachers to effectively impart knowledge and inspire students' curiosity. They should possess a deep understanding of the subject, stay updated with the latest developments, and be able to answer students' questions confidently.\n\n4. Adaptability: Successful teachers are flexible and adaptable, as each student has unique needs and learning styles. Teachers should be able to adjust their teaching methods, materials, and assessments to accommodate different learning preferences and address individual strengths and challenges.\n\n5. Patience and Empathy: Teaching can be challenging, and patience is crucial when working with students of varying abilities and backgrounds. Teachers need to empathize with their students, be understanding of their struggles, and provide emotional support when needed. Building positive relationships based on trust and respect is important for creating a supportive learning environment.\n\nWhile these skills are critical for teacher success, it is important to note that additional skills like creativity, organization, problem-solving, and technological proficiency also enhance teaching effectiveness." }, "finish_reason": "stop" } ], "usage": { "prompt_tokens": 21, "completion_tokens": 287, "total_tokens": 308 } }

ERROR OCCURED:

image

eco-bone commented 1 year ago

The first api call to openai API is working but the second call is not working. I am getting the original paragraph but after different ways of testing, the second api call is giving this error no matter what the input is.:

org.springframework.web.client.HttpClientErrorException$BadRequest: 400 Bad Request: "{ "error": { "message": "We could not parse the JSON body of your request. (HINT: This likely means you aren't using your HTTP library correctly. The OpenAI API expects a JSON payload, but what was sent was not valid JSON. If you have trouble figuring out how to fix this, please contact us through our help center at help.openai.com.)", "type": "invalid_request_error", "param": null, "code": null }}"

TRYING TO SEND A REQUEST WITH LESS NUMBER OF TOKENS TO TEST IF THE MESSAGE IS TOO LONG FOR THE API CALL

eco-bone commented 1 year ago

Update on the JSON error:

I tried the following function instead of inputting the entire paragraph and it worked:

String para2 = chatService.getChatCompletion("Hello");

OUTPUT:

image

I am getting the response "Hello! How can I assist you today?" but not getting an output when I am giving the paragraph as input.

TEST OTHER KINDS OF INPUTS. HAVE TO FIND A WAY TO GIVE THE PARAGRAPH AS THE INPUT

eco-bone commented 1 year ago

http://localhost:9090/automation/createDataset/1

eco-bone commented 1 year ago

The problem was occurring due to the prompt for the second being in the wrong format. When put in the following format, the prompt ran successfully:

image

The problem to fix now is that java by default returns lengthy strings in the format which contain "\n". Have to figure out how to remove it.

eco-bone commented 1 year ago

https://platform.openai.com/docs/guides/text-generation/chat-completions-api

eco-bone commented 1 year ago

Changed the function to get the original and paraphrase using one prompt. It is using almost 600 less tokens including both output and input. Have to figure out how to slice the output and save them separately in original and paraphrase.

eco-bone commented 12 months ago

Finally got the dataset creation method working. Now need to work around on the rate limit provided by openAI API.

this is the output gotten when trying to put 4 requests at once:

image

One solution is to use Thread.sleep so that the thread waits for few seconds after every 3 api calls so there is no error thrown. Unfortunately pausing the thread is considered as bad code and should be used only as the last resort.

An alternative to this is to use a different ai api.

eco-bone commented 12 months ago

https://status.openai.com/

check the status of openAI API for downtime

eco-bone commented 12 months ago

New solution: We have 20,000 tokens per minute limit. The ballpark for every request would be around 100 tokens for every request and 1200 for every dataset object. If i take 15 dataset objects for every request, for 200 requests per account for 2 accounts; it will give me 15x200x2 number of dataset objects everyday which is equal to 6000 per day. If I run it for 10 days, it will come to be 60,000 dataset objects.

eco-bone commented 12 months ago

Target ds objects in the next 15 days: 50,000

eco-bone commented 12 months ago

the perfect prompt for individual :

Provide the top 5 skills required to excel in a specific profession.Describe each skill in around 100 words. Then, paraphrase this information to about 10 percent variation, but maintain the length of each skill description. Avoid using headings and separate the original and paraphrased versions using '==' as a delimiter. Ensure the retention of the five skills and their respective descriptions' length.

OUTPUT

Skill 1: Communication In any profession, effective communication is paramount. This skill involves not only the ability to articulate ideas clearly but also to actively listen and comprehend others. It encompasses written, verbal, and non-verbal forms of interaction. A professional who excels in communication can convey complex concepts in an understandable manner, resolve conflicts, and build strong relationships. Clear communication fosters collaboration, boosts team morale, and ensures that everyone is aligned towards common goals.

Skill 2: Adaptability The capacity to adapt to changing circumstances is crucial for success in any field. Being adaptable means embracing change with an open mind, adjusting to new technologies, methods, or situations swiftly and efficiently. Professionals who excel in adaptability showcase resilience, innovation, and a willingness to learn. They can navigate uncertainties, embrace diverse perspectives, and quickly pivot strategies to achieve desired outcomes, even in unpredictable environments.

Skill 3: Problem-solving Proficiency in problem-solving is indispensable in any profession. It involves the ability to identify issues, analyze situations, and develop effective solutions. Individuals skilled in this area display critical thinking, creativity, and resourcefulness. They approach challenges methodically, breaking them down into manageable components to derive innovative and practical resolutions. Exceptional problem solvers foresee potential obstacles, devise contingency plans, and contribute significantly to organizational growth through their adept handling of complex issues.

Skill 4: Leadership Leadership is not solely about managing teams; it involves inspiring, guiding, and motivating others towards a common vision. Proficient leaders exhibit integrity, empathy, and the ability to delegate tasks efficiently. They foster an environment of trust, encourage collaboration, and empower team members to reach their full potential. Effective leaders communicate a clear direction, make sound decisions, and serve as role models, driving organizational success through their strategic vision and people-centric approach.

Skill 5: Time Management The skill of managing time efficiently is crucial in any profession. It involves prioritizing tasks, setting realistic deadlines, and allocating resources judiciously. Professionals adept in time management maximize productivity, minimize procrastination, and deliver high-quality work consistently. They employ strategies like goal setting, task batching, and eliminating distractions to optimize their workflow. Effective time management ensures meeting deadlines, reduces stress, and allows professionals to maintain a healthy work-life balance.

==

Skill 1: Communication Effective communication is fundamental in any field. It goes beyond articulating thoughts clearly to actively listening and understanding others. This skill covers various forms of communication, including written, verbal, and non-verbal. A professional who excels here can simplify complex ideas, resolve conflicts, and build robust relationships. Good communication fosters collaboration, boosts team spirit, and ensures everyone is working towards shared objectives.

Skill 2: Adaptability The ability to adapt to changing circumstances is vital. Being adaptable means embracing change, swiftly adjusting to new technologies, methods, or situations. Those who excel here show resilience, innovation, and a hunger for learning. They navigate uncertainties, embrace diverse perspectives, and quickly pivot strategies, even in unpredictable environments.

Skill 3: Problem-solving Proficiency in problem-solving is indispensable. It involves identifying issues, analyzing situations, and devising effective solutions. Skilled individuals exhibit critical thinking, creativity, and resourcefulness. They break down challenges into manageable components to find innovative and practical resolutions.

Skill 4: Leadership Leadership is more than managing teams; it’s about inspiring, guiding, and motivating others. Proficient leaders show integrity, empathy, and efficient delegation. They foster trust, encourage collaboration, and empower team members to reach their potential.

Skill 5: Time Management Efficient time management is critical. It involves prioritizing tasks, setting realistic deadlines, and using resources wisely. Professionals skilled in this area maximize productivity, minimize procrastination, and consistently deliver high-quality work. Effective time management ensures meeting deadlines and maintaining work-life balance.

eco-bone commented 12 months ago

api is giving only 1 output even though i am asking for more. openAI may have put measures in place to only give output to a fixed number of queries in one request. have to confirm still testing.

eco-bone commented 12 months ago

prompt:

Provide the top 5 skills required to excel as a/an [profession].Describe each skill in around 150 words. Then, paraphrase this information to about 10 percent variation, but maintain the length of each skill description. Avoid using headings and separate the original and paraphrased versions using '==' as a delimiter. Ensure the retention of the five skills and their respective descriptions' length. give the output for this for a teacher,biologist and driver.Ensure consistent formatting throughout the responses for each profession, regardless of the content or profession mentioned.

OUTPUT:

Profession: Teacher

  1. Communication Skills: Effective communication is essential for teachers as they need to convey information clearly and engage students in the learning process. Teachers must have strong verbal and written communication skills to explain complex concepts, give instructions, and facilitate discussions. Additionally, they should be good listeners to understand students' needs and provide appropriate support and feedback.

  2. Classroom Management: Successful teachers possess excellent classroom management skills to maintain a productive and orderly learning environment. They must establish and enforce rules, maintain discipline, and address conflicts effectively. Additionally, teachers need to manage time efficiently, ensuring that lessons are well-structured and engaging without distractions or disruptions.

  3. Subject Matter Expertise: Teachers need to have a deep understanding of the subjects they teach to effectively impart knowledge and inspire students. They should be knowledgeable in their area of expertise, continually update their knowledge, and have the ability to explain concepts in multiple ways to accommodate diverse learning styles.

  4. Adaptability: Flexibility and adaptability are crucial traits for teachers as each student is unique and may require different teaching approaches. Teachers should be willing to modify their instructional methods, adjust lesson plans, and use various resources to meet students' individual needs. Moreover, being adaptable allows teachers to embrace changes in educational policies and technologies.

  5. Patience and Empathy: Patience and empathy are vital attributes for teachers to create a supportive and nurturing classroom environment. Teachers should remain patient when encountering difficulties or when students face challenges in grasping concepts. They should also demonstrate empathy by understanding students' perspectives, providing emotional support, and promoting inclusivity to ensure all students feel valued and respected.

==================================================================================================== Paraphrased Version:

  1. Communication Skills: Teachers should possess strong communication abilities, both verbally and in writing, to effectively convey information and engage students in the learning process. This includes explaining complex concepts, offering instructions, and facilitating discussions. Additionally, active listening is crucial for teachers to understand students' needs and offer appropriate support and feedback.

  2. Classroom Management: Excellent classroom management skills are key for teachers to create an orderly and productive learning environment. This includes establishing and enforcing rules, effectively dealing with conflicts, and maintaining discipline. Moreover, teachers must manage time efficiently, ensuring well-structured and engaging lessons that minimize distractions.

  3. Subject Matter Expertise: Teachers must have a deep understanding of the subjects they teach to effectively impart knowledge and inspire students. They should continuously update their knowledge and be capable of explaining concepts in diverse ways suitable for various learning styles.

  4. Adaptability: Adaptability is a critical trait for teachers as each student is unique and may require different teaching approaches. Teachers should be open to modifying their instructional methods, adjusting lesson plans, and utilizing various resources to meet individual student needs. Additionally, being adaptable enables teachers to embrace changes in educational policies and technologies.

  5. Patience and Empathy: Teachers should possess patience and empathy to cultivate a supportive and nurturing classroom environment. Demonstrating patience during difficulties or when students struggle with understanding concepts is crucial. Furthermore, teachers should empathize with students, understand their perspectives, offer emotional support, and promote inclusivity to ensure every student feels valued and respected.

====================================================================================================

Profession: Biologist

  1. Scientific Knowledge: A strong foundation in scientific knowledge is essential for biologists. They need to have a deep understanding of biology concepts, terminology, and theories. This includes knowledge of various branches of biology, such as genetics, ecology, and biochemistry. Biologists should be constantly updated with the latest advancements in their field to stay relevant and contribute effectively.

  2. Research Skills: Biologists must possess excellent research skills to design and conduct experiments, analyze data, and draw meaningful conclusions. They should be proficient in laboratory techniques, data collection, statistical analysis, and scientific methodologies. This enables them to contribute to scientific discoveries, publish research papers, and collaborate with other scientists.

  3. Critical Thinking: Biologists need strong critical thinking skills to analyze complex problems, evaluate information, and develop innovative solutions. This involves the ability to identify patterns, draw logical connections, and think creatively. Biologists often encounter complex biological phenomena that require critical thinking skills to understand and explain.

  4. Attention to Detail: Precision and attention to detail are crucial in biology to ensure accurate results and minimize errors. Biologists should possess a meticulous approach when conducting experiments, recording data, and analyzing samples. Attention to detail also extends to documentation and following safety protocols to maintain integrity and reliability in research.

  5. Communication and Collaboration: Effective communication and collaboration skills are vital for biologists to share research findings, present scientific papers, and collaborate with peers. Biologists need to communicate complex scientific concepts in a clear and concise manner, both in oral and written forms. They should also collaborate effectively with multidisciplinary teams, fostering an environment of knowledge exchange and cooperation.

==================================================================================================== Paraphrased Version:

  1. Scientific Knowledge: Biologists must possess a comprehensive understanding of biology concepts, terminology, and theories. This encompasses various branches of biology, such as genetics, ecology, and biochemistry. It is crucial for biologists to remain updated with the latest advancements in their field to contribute effectively.

  2. Research Skills: Strong research skills are critical for biologists to design and conduct experiments, analyze data, and derive meaningful conclusions. Proficiency in laboratory techniques, data collection, statistical analysis, and scientific methodologies enables them to make scientific discoveries, publish research papers, and collaborate with other scientists.

  3. Critical Thinking: Biologists require robust critical thinking abilities to analyze complex problems, evaluate information, and generate innovative solutions. This involves identifying patterns, establishing logical connections, and thinking creatively. Critical thinking is essential when tackling intricate biological phenomena that demand comprehension and explanation.

  4. Attention to Detail: Precision and attention to detail are vital in biology to ensure accurate results and minimize errors. Biologists adopt a meticulous approach when conducting experiments, recording data, and analyzing samples. They also prioritize documentation and follow safety protocols to uphold research integrity and reliability.

  5. Communication and Collaboration: Effective communication and collaboration skills are crucial for biologists to share research findings, present scientific papers, and collaborate with peers. Biologists must communicate complex scientific concepts clearly and concisely, both orally and in writing. Additionally, they need to foster efficient collaboration with multidisciplinary teams, promoting knowledge exchange and cooperation.

====================================================================================================

Profession: Driver

  1. Excellent Driving Skills: As a driver, possessing excellent driving skills is paramount. This includes expertise in vehicle control, knowledge of traffic rules, maneuvering techniques, and road safety. A skilled driver should be able to handle different road and weather conditions, anticipate potential hazards, and react swiftly to ensure the safety of passengers and other road users.

  2. Defensive Driving: Defensive driving is an essential skill that enables drivers to anticipate and avoid potential accidents. It involves staying alert, maintaining appropriate speeds, and keeping a safe distance from other vehicles. Additionally, understanding the behavior of other drivers and predicting their actions contributes to being a defensive driver.

  3. Patience and Stress Management: Patience and stress management are crucial skills for drivers, especially when encountering traffic congestion, unruly drivers, or challenging road situations. Having the ability to remain calm, patient, and composed is vital for making rational decisions and avoiding aggressive behaviors that could jeopardize safety.

  4. Navigation Skills: Drivers should possess good navigation skills to efficiently plan routes, read maps, and effectively utilize navigation tools. Being familiar with local streets, landmarks, and traffic patterns aids in choosing the most efficient routes and minimizing travel time.

  5. Mechanic Knowledge: A basic understanding of vehicle mechanics is valuable for drivers. This includes knowledge of routine vehicle maintenance, identifying common vehicle issues, and performing simple repairs when necessary. Being able to address minor vehicle problems on-the-go can prevent inconveniences and delays.

==================================================================================================== Paraphrased Version:

  1. Excellent Driving Skills: Having exceptional driving skills is crucial for drivers. This encompasses mastering vehicle control, being knowledgeable about traffic rules, adept at maneuvering techniques, and prioritizing road safety. Skillful drivers must be capable of handling various road and weather conditions, foreseeing potential hazards, and responding promptly to ensure the safety of passengers and others on the road.

  2. Defensive Driving: Defensive driving is a vital skill that enables drivers to anticipate and avoid potential accidents. It involves maintaining attentiveness, adhering to appropriate speeds, and maintaining a safe distance from other vehicles. Additionally, understanding the behavior of other drivers and forecasting their actions contributes to becoming a defensive driver.

  3. Patience and Stress Management: Drivers should possess patience and stress management skills, especially when faced with traffic congestion, unruly drivers, or challenging road situations. The ability to remain calm, patient, and composed is crucial for making informed decisions and avoiding aggressive behaviors that may compromise safety.

  4. Navigation Skills: Proficiency in navigation skills allows drivers to plan routes efficiently, read maps, and effectively use navigation tools. Being familiar with local streets, landmarks, and traffic patterns assists in selecting the most efficient routes and minimizing travel time.

  5. Mechanic Knowledge: Drivers benefit from having basic knowledge of vehicle mechanics. This includes understanding routine vehicle maintenance, identifying common vehicle issues, and performing minor repairs when necessary. The capability to address minor vehicle problems on-the-go helps prevent inconvenience and delays.

====================================================================================================

eco-bone commented 12 months ago

Provide the top 5 skills required to excel in a specific profession.Describe each skill in around 100 words. Then, paraphrase this information to about 10 percent variation, but maintain the length of each skill description. Avoid using headings and separate the original and paraphrased versions using '==' as a delimiter. Ensure the retention of the five skills and their respective descriptions' length. Do it in the following format:

Skill 1: 100 words Skill 2: 100 words Skill 3: 100 words Skill 4: 100 words Skill 5: 100 words

Skill 1: 100 words Skill 2: 100 words Skill 3: 100 words Skill 4: 100 words Skill 5: 100 words

ADD THE DOUBLE EQUALS TO DELIMITER WHEN PUTTING IN AS PROMPT.

this prompt works for gpt-3.5-turbo-16k

eco-bone commented 12 months ago

2023-11-21T17:24:50.820+05:30 ERROR 30456 --- [nio-9090-exec-1] c.d.a.controller.AutomationController : 503 Service Temporarily Unavailable: "{"error":{"code":503,"message":"Service Unavailable.","param":null,"type":"cf_service_unavailable"}}"

Error occurred multiple times need to create retry eventually.

eco-bone commented 12 months ago

open ai access over for first account.......

eco-bone commented 12 months ago

final prompt decided:

Explain the top 5 skills required to be an outstanding teacher in about 500 words. Then paraphrase the entire 500 words while retaining the numbered points. Don't leave lines anywhere. If one point ends, start the next point from the very next line. Separate the original and paraphrase using a delimiter '=='."

attempts to create multiple dataset entries in one prompt halted for now.

poa is to move forward with 1 total entry per api call. this will be more expensive and take a little more time though.

eco-bone commented 12 months ago

2023-11-24T12:36:59.682+05:30 INFO 13536 --- [nio-9090-exec-2] c.d.a.service.AutomationServiceImpl : moving to next profession... 2023-11-24T12:36:59.973+05:30 ERROR 13536 --- [nio-9090-exec-2] c.d.a.service.AutomationServiceImpl : Error occured while making api call..... 2023-11-24T12:36:59.973+05:30 ERROR 13536 --- [nio-9090-exec-2] c.d.a.service.AutomationServiceImpl : 429 Too Many Requests: "{ "error": { "message": "Rate limit reached for gpt-3.5-turbo in organization org-QmbDTx5oOJWjknL8V0KVCmVL on requests per min (RPM): Limit 3, Used 3, Requested 1. Please try again in 20s. Visit https://platform.openai.com/account/rate-limits to learn more. You can increase your rate limit by adding a payment method to your account at https://platform.openai.com/account/billing.", "type": "requests", "param": null, "code": "rate_limit_exceeded" }}" 2023-11-24T12:36:59.973+05:30 INFO 13536 --- [nio-9090-exec-2] c.d.a.service.AutomationServiceImpl : moving to next profession...