VolhaBazhkova
commented
8 months ago It looks like it's the same issue as #160
Closed WVd3ilia closed 3 weeks ago
VolhaBazhkova
commented
8 months ago It looks like it's the same issue as #160
avavilau
commented
8 months ago @WVd3ilia please make sure that there is no private information disclosed above. EPAM github Issue tracker is public.
WVd3ilia
commented
3 weeks ago Is not reproducible now with steps in https://github.com/epam/ai-dial-chat/issues/160
Model: GPT-4-Turbo-1106
Same error when use RAG and attach table as Excel file.
The error started to appear after 10+ successful generations with tests of different prompts and different input files. My colleague gets the same error after the 1st try of my prompt.
Prompt:
As a technology and process improvement expert, your experience and insight are invaluable for enhancing our organization's operational efficiency. Please construct a concise executive summary with overall state and key finding, and develop a suite of 3-5 specific and actionable recommendations for each dimension identified from the maturity assessment. Recommendations should be designed to address the challenges observed within dimensions that garnered the lowest maturity scores with the highest importance weights especially when have direct impacts on project delivery, client satisfaction, or revenue generation. Make sure to align recommendations with the best practices and standards within our industry or relevant field, ensuring that our strategies adhere to established benchmarks for excellence.
Your insight will not only guide our immediate improvements but also inform our approach for sustained growth and development. We look forward to your expert recommendations, which will advance our project management practices effectively.
Note: Please integrate the provided expert recommendations and maturity assessment results as references for your detailed analysis and tailored advice.
Dimension Sub-dimension Sub-sub dimension Statement Total Avg Score Assessor_1 Assessor_2 Assessor_3 Assessor_3
Governance and Communication (weight 1.0) 3.6 3.6 3.6
Governance and Communication (weight 1.0) The Customer regularly fills in surveys, results are analyzed, corrective actions are taken (weight 1.0) 4.0 4 SUSTAINABLE The recent CSAT campaign revealed good results (no distractors among the project stakeholders). No custom survey with a shorter feedback loop has been introduced so far. It is worth considering the implementation of a custom survey that offers shorter feedback loops. This approach allows for more frequent monitoring of customer satisfaction levels, facilitating prompt adjustments and enhancing the overall customer experience. 4 SUSTAINABLE no surveys. feedback is gathered verbally on 1on1s. action points are transferred to the team and tracked in weekly status updates
Governance and Communication (weight 1.0) Regular 1-2-1 meetings with the Customer are conducted (weight 1.0) 5.0 5 TRANSFORMATIVE Project Manager has a bi-weekly meeting with Head of Engineering, the primary project stakeholder from the client side. Additionally, Delivery Manager and Account Manager maintain regular contact with a broad range of stakeholders on the client side. 5 TRANSFORMATIVE scheduled for every week + on demand with all major stakeholders
Governance and Communication (weight 1.0) Regular 1-2-1 meetings with team members are conducted (weight 1.0) 4.0 4 SUSTAINABLE Project Manager conducts monthly 1-2-1 meetings with the key people on the project, while conversation with the rest of the team is delegated to Tech Lead. The outcomes and insights from these 1-2-1 meetings are not consistently captured in Teams DeliveryCentral. It is recommended to make Teams DeliveryCentral updates on monthly basis to ensure alignment with EPAM standards and facilitate effective information exchange with RMs and HRs 4 SUSTAINABLE New Project Coordinator just kicked off his first iteration of one2ones, need to continue
Governance and Communication (weight 1.0) Communication plan is documented, aligned with RACI and shared with the team (weight 1.0) 2.5 2 DEVELOPING While the established communications align with the chosen delivery framework, there is a need to formalize them through a written communication plan. This will ensure that the communication processes, channels, and stakeholders are clearly documented and communicated to the team, enabling effective and consistent communication throughout the project. 3 ENGAGING no. communication plan is well understood and implemented between AM, DM & Project Coordinator. need to write down
Governance and Communication (weight 1.0) Roles and responsibilities are defined and documented (weight 1.0) 2.5 3 ENGAGING The current RACI matrix for role and responsibility assignment within the team lacks clarity, as some activities have multiple assignees and others have no assigned responsibility or accountability. It is crucial to review and update the RACI matrix to address the issue of blurred responsibilities and ensure clear ownership for each activity. 2 DEVELOPING no. R&R are well understood and implemented between AM, DM & Project Coordinator. need to write down. There is clear disconnect on Tech Lead roles and responsibilities. Looking for a new Tech Lead
Staffing and People Management (weight 1.0) 2.8 3.0 2.7
Staffing and People Management (weight 1.0) Resource plan is in place and updated regularly (weight 1.0) 5.0 5 TRANSFORMATIVE The resource plan is diligently maintained within the Staffing Desk. 5 TRANSFORMATIVE in staffing desk
Staffing and People Management (weight 1.0) Knowledge transfer procedure for newcomers is in place and used (weight 1.0) 2.5 3 ENGAGING The current onboarding documentation provides basic information about accounts configuration and tool links, but lacks comprehensive details on project governance, processes, and policies. To ensure a smooth and comprehensive onboarding experience, it is recommended to enhance the documentation by including information about roles and responsibilities, communication plan, Definition of Ready (DoR), Definition of Done (DoD), engineering standards, vacation and sick leave policies, time reporting policies, and other relevant details. 2 DEVELOPING in customer's confluence. outdated. require review and update
Staffing and People Management (weight 1.0) HR Analytics (Attrition, Motivation, Performance) is collected and analyzed, corrective actions are taken (weight 1.0) 2.0 2 DEVELOPING While the attrition rate is currently low, it is crucial to have a deep understanding of the group dynamics and consistently gather feedback to ensure sustainable retention rates. Project Manager hasn't been informed about the results of the latest EPAM People Campaign. It is strongly advised to establish communication with the HR unit to address this gap. Besides, relying only on global EPAM People campaigns, though valuable, may not bring frequent enough insights into the project's unique dynamics. Therefore, it is highly recommended to supplement these global campaigns with custom quarterly surveys. These surveys will allow for regular measurement of the team's satisfaction levels, identification of any concerns, and timely intervention as needed. 2 DEVELOPING Previous Project Coordinator were not tracking feedback and performance evaluation in Delivery Central. New PC need to establish this competency
Staffing and People Management (weight 1.0) Staffing roles and responsibilities are defined, key contact points are documented and shared with the leadership team (weight 1.0) 4.5 4 SUSTAINABLE Project Manager, with her comprehensive understanding of the staffing process, takes charge of the staffing activities. She is well-informed about the contact points on the supply side and can commit to the timely processing of staffing requests. It is recommended to document the staffing governance and process to facilitate a streamlined and effective approach in case of rapid ramp up with huge number of staffing requests at a time. 5 TRANSFORMATIVE in staffing desk
Staffing and People Management (weight 1.0) Succession planning for key roles is in place (weight 1.0) 1.0 1 OPPORTUNISTIC Currently, there is no practice of succession planning in place. To address this gap, it is recommended to establish formalized succession plans for key project roles. These plans should include identifying suitable individuals to assume critical positions and providing them with proper training and mentoring opportunities. By implementing structured succession plans, the project can proactively mitigate risks associated with talent gaps, and ensure a smooth transition when key team members leave or move on to new roles. 1 OPPORTUNISTIC no. just finished rotation of Project Coordinator. Preparing for rotation of Tech Lead
Staffing and People Management (weight 1.0) There are tools and practices to keep team morale and motivation on a high level (Town Halls, Team Health Surveys, Recognition Program) (weight 1.0) 2.0 3 ENGAGING Currently people retention and motivation approach is limited by kudos given at retrospectives and via Heroes portal. It is recommended to establish close collaboration between Project Manager and the HR unit to implement an advanced people retention strategy. This strategy may include initiatives such as TownHalls, appreciation digests, branded heroes badges, well-being campaigns, and online team-building activities. 1 OPPORTUNISTIC no. using only spot bonuses and direct feedback after Sprint Demos.
Process Management (weight 1.0) 3.0 3.7 2.3
Process Management (weight 1.0) Change management procedure is documented, approved, and followed (weight 1.0) 3.0 3 ENGAGING When changes occur, Project Manager addresses their impact during Scrum of Scrums events and facilitates decisions regarding project plan updates. However, there is no formalized change management process with an appropriate log in place. It is recommended to establish a structured change management process that includes a change log to effectively track and manage project changes, ensuring transparency and accountability. 3 ENGAGING Close to Not-Applicable. As team and customer works in Scaled Agile and Scrum teams are used, the change management is handled by continuous refinement of backlog. this is documented. The problem is in embracing the change by the team. When critical change in scope was identified which would require significant shift in technical approach, it was close to impossible to get the team onboarded and shift to new approach.
Process Management (weight 1.0) Production release, hot-fix release, and roll-back procedures are documented, approved and shared with the team (weight 1.0) 3.5 3 ENGAGING The current release documentation offers limited information on the release process, lacking a comprehensive and visual representation of the release flow along with clear definitions of each step and the responsible roles. It is recommended to enhance the documentation by incorporating a detailed and visually represented release flow that outlines the steps and roles involved, providing a better understanding and guidance for effective release management. 4 SUSTAINABLE somehow. there is documentation in customer's confluence. but it should be reviewed with new Project Coordinator and team
Process Management (weight 1.0) Continuous improvement events / procedures are defined and being performed (weight 1.0) 3.0 5 TRANSFORMATIVE Retrospectives are an integral component of the communication plan. During these sessions, actions that are agreed upon are recorded, and their progress is then tracked in subsequent meetings. 1 OPPORTUNISTIC retrospective in the team in past was often omitted. the action items after retrospective were not solved. feedback gathered in retrospectives was pretty shallow. other continuous improvement events does not exist
Process Management (weight 1.0) Project milestones and / or end dates are defined (weight 1.0) 2.0 3 ENGAGING The project plan currently consists of short-term milestones, typically covering a one to two-month timeframe. However, long-term planning is not in place, leading to situations where the team completes their work before new scope is defined. It is recommended to collaborate with the client to establish a long-term planning process to ensure a continuous flow of work and avoid idle time for the team. 1 OPPORTUNISTIC they were, but team had issues with adopting/tracking/adjusting them to current state. this is one of priorities of new Project Coordinator
Process Management (weight 1.0) A specific estimation approach is picked, documented and used by the whole team (weight 1.0) 3.5 5 TRANSFORMATIVE The team employs relative estimation using story points, following best practices where all team members participate to produce a single estimate aligned with the Fibonacci sequence. The story point baseline is established and documented. 2 DEVELOPING On strategic initiatives level there is customer's framework for estimations. on day by day team tasks estimations are opportunistic and could be improved. this is on new Project Coordinator radar.
Process Management (weight 1.0) Ticket flow and ticket board are documented and shared with the team (weight 1.0) 2.5 2 DEVELOPING The team currently lacks documentation regarding the ticket flow and ticket board, which can lead to inconsistencies in ticket management practices. It is recommended to create comprehensive documentation that clearly defines each status in the flow, outlines the possible transitions, and identifies the responsible roles for these transitions. 3 ENGAGING Jira is used. it might evolve with help of new Project Coordinator
Process Management (weight 1.0) Definition of Ready and Definition of Done are documented and shared with the team (weight 1.0) 3.5 5 TRANSFORMATIVE The team has established and documented clear Definition of Ready (DoR) and Definition of Done (DoD) criteria, which are well understood by all team members. 2 DEVELOPING It exists, it is ambitious, but team rarely meets all the requirements from DoR or DoD. This is one of priorities to tackle by new Project Coordinator
Reporting (weight 1.0) 2.8 3.0 2.7
Reporting (weight 1.0) Delivery metrics collecting and analysis are set. Action log based on regular metrics review is maintained (weight 1.0) 2.0 3 ENGAGING The client has recently introduced a regular review of productivity metrics, including Cycle Time, Velocity, Burndown, and the number of Carried Over items, collected through JIRA. However, these metrics alone may not provide a comprehensive view of the delivery state. To gain a more holistic understanding, it is recommended to incorporate additional metrics into the evaluation process to cover other delivery aspects such as scope and quality state. It is also important to establish clear KPIs for each metric to ensure they are effectively measured and tracked. Project Manager should actively record all cases where the team misses the established KPIs and maintain an action log. This will enable a proactive approach to addressing any performance gaps and taking necessary corrective actions to improve delivery outcomes. Additionally, it is recommended to integrate the metrics into the client reporting template. 1 OPPORTUNISTIC sprint capacity vs velocity is tracked. no other metrics are in place. analysis of velocity is not done.
Reporting (weight 1.0) RAID (Risks, Assumptions, Issues and Dependencies) register is in place and updated regularly (weight 1.0) 3.0 3 ENGAGING The internal RAID log in DeliveryCentral is not properly maintained and contains outdated items. Delivery Manager is advised to maintain the RAID log in DeliveryCentral to provide EPAM senior management with a comprehensive understanding of the project's status and any mitigation actions that have been undertaken. The client report includes a risks and issues section, but it lacks essential details such as probability and impact assessment, mitigation strategies, assigned owner, and due dates. It is recommended to enhance the log template to capture these critical elements, enabling better management and mitigation of risks and issues within the project. 3 ENGAGING yes. weekly status update is used for that. going to start using delivery central to keep track of it
Reporting (weight 1.0) Status reports are regularly sent and contain the project RAG with rationale, milestones status, key risks, and dependencies (weight 1.0) 3.5 3 ENGAGING The frequency of updates provided to the DC Health dimensions is inconsistent and does not adhere to EPAM standards. The overall project status is not aligned with specific dimensions, and Amber statuses lack a proper Go-to-green plan with clear actions, owners, and due dates. Additionally, the update in DC lacks crucial information about the project roadmap, which is essential for providing senior management with visibility into completed deliverables and upcoming solutions. It is recommended to conduct a more comprehensive evaluation of the RAG status of the project. This evaluation should include providing transparent information about the RAG rationale and a clear go-to-green plan. Additionally, it is important to incorporate the project roadmap as an integral part of the update. The client reporting is conducted weekly using a well-structured template that includes RAG status per core dimension, information about completed work and plans for the next week, a risks and issues log, and a milestones view. However, there are still minor adjustments that should be made. It is recommended to enhance the RAID section by including the details mentioned above, and also to add the status of milestones (on track, at risk, delayed) to the milestones view. 4 SUSTAINABLE yes
Contract and Budget Tracking (weight 1.0) 4.0 3.3 4.6
Contract and Budget Tracking (weight 1.0) Delivery manager has access to finance-related information (weight 1.0) 3.5 2 DEVELOPING Project Manager receives limited financial updates from Account Manager, with only budget spending details being communicated and no disclosure of the project margin. The decision to address this gap lies with the account leadership, considering the maturity of Project Managers. 5 TRANSFORMATIVE Yes
Contract and Budget Tracking (weight 1.0) Budget spending is tracked (weight 1.0) 5.0 5 TRANSFORMATIVE The responsibility for budget spending lies within the scope of Account Manager. Project Manager receives updates on the current status of budget expenditures during regular interactions with Account Manager. 5 TRANSFORMATIVE yes
Contract and Budget Tracking (weight 1.0) The Commercial model matches the engagement model. Both are adequate in the scope of the whole contract and match our responsibilities and level of ownership (weight 1.0) 4.0 4 SUSTAINABLE Commercial model is T&M but engagement model is T&M with deliverables. The team structure includes Project Coordinator who is responsible for project management areas like roadmapping and risk management. it is Account Strategy to showcase for the customer grater value provided by EPAM, as we are competing with other vendor.
Contract and Budget Tracking (weight 1.0) P&L simulation is a part of the staffing process, seniority changes and rate card review (weight 1.0) 5.0 5 TRANSFORMATIVE Project Manager requests running of a P&L simulation prior to each staffing decision to ensure a thorough evaluation of the financial impact. 5 TRANSFORMATIVE yes
Contract and Budget Tracking (weight 1.0) Delivery manager has read the contract and understands the terms (weight 1.0) 2.5 1 OPPORTUNISTIC As the respondent (Melissa Calderon) does not have access to the contract information, it is recommended to provide her with access to ensure that Project Manager has a comprehensive understanding of the obligations of all parties involved. 4 SUSTAINABLE DM knows the contract. This comment was misunderstanding. Probably it referred to Project Coordinator.