search

mikeb1 / harikg

0 stars 0 forks source link

Google Meet with Hari 12222023 - Mainly KG walkthrough #1

Open mikeb1 opened 9 months ago

mikeb1 commented 9 months ago

https://drive.google.com/file/d/1wfV82KgNEpmhl0uyTbfK5lDcg2J-VFux/view?usp=sharing

Please act as a professional note taker and create an outline for the topics of the conversation below. Be as detailed as possible:

yws-dunr-jmg (2023-12-22 09:36 GMT+2) - Transcript
Attendees
Harikrishnan V K, Mike Belson, Mike Belson's Presentation
Transcript
This editable transcript was computer generated and might contain errors. People can also change the text after it was created.
Harikrishnan V K: The URLs up very long really.
Mike Belson: I'm sorry Can you say that again?
Harikrishnan V K: No noise, you were online really?
Mike Belson: yeah, you saw that so yeah, it's an issue for some reason or bot isn't loading now. And a lot of websites, I'm not really sure what the reason is.
Harikrishnan V K: a lot of applications are
Harikrishnan V K: so I didn't understand.
Mike Belson: sit Yeah, I'm saying the bot or…
Harikrishnan V K: he Who?
Mike Belson: but isn't loading and a lot of websites now.
Harikrishnan V K: Okay.
Mike Belson: So that's a major issue. And obviously something like this should never happen. And it's been working pretty smoothly.
Harikrishnan V K: Yeah.
Mike Belson: Until yesterday at night when we got this alert. So hopefully Ivan and Bogdan can resolve this soon this morning. it's loading. and it's not
Mike Belson: You can see my screen, right?
Harikrishnan V K: Yeah, I can see the controllers.
Mike Belson: Yeah, so yeah, so that's an issue. All right, and there's a lot of websites now that…
Harikrishnan V K: Yeah.
Mike Belson: because of that for example this website over here. Is also not loading. So none of the website seems to seem to be loading now correctly. Also, you can see here that there's a little bit of a styling issue. That it's wrapping to another line. That also shouldn't happen. I mean our styling should be immune from the hosting websites Style. right
Harikrishnan V K: is this something that the WordPress change or it was any update happened from your
Mike Belson: That I'm not sure I don't think there was any new deployments. And sometimes when this happens temporary and you can see that openai, I'm might be problematic but in this case. That doesn't seem to have been the issue. Let's see.
Harikrishnan V K: yeah, and…
Mike Belson: It says no.
Harikrishnan V K: so messages was showing some WordPress some as
Mike Belson: Some WordPress why?
Harikrishnan V K: in the console if you can see, it's WP adminus that you're seeing.
Mike Belson: In the council over here.
Harikrishnan V K: You Okay one second. The website link is let's see. he life Can you give me this link? so that I can just take yeah.
Mike Belson: Yeah, but I don't think there's gonna be much for you to be able to see here.
Harikrishnan V K: I know I can do much without the actual.
Mike Belson: The thing is if you give Eva on your IP, then he can show you a lot more information in the console. Yes, you see information like this.
Harikrishnan V K: Okay.
Mike Belson: widget instant books And all the stuff here, I don't think you can see this, right?
Harikrishnan V K: Give me one second. Let me just take no, I can't see that.
Mike Belson: Yeah, so he exposes this based on the IP. so that we are.
Harikrishnan V K: Nobody available that it will be for me or so.
Mike Belson: It is. So certain things won't be available certain things are certain things, aren't I guess? Okay, so what I would like to do. Is just go over our priorities what we discussed we started to discuss yesterday.
Mike Belson: I want to kind of take things to the top. So that we're on the same page…
Harikrishnan V K: Okay.
Mike Belson: what our objectives are, okay.
Mike Belson: So objective number one is you can see this URL. I've shown you how this works, right?
Harikrishnan V K: Yes, show me that.
Mike Belson: Okay, so You've also seen in all the data that we collect, correct.
Harikrishnan V K: That's true.
Mike Belson: Okay That data is stored in firestore, but it's not stored in. the knowledge graph Okay.
Mike Belson: so that data we use it to see that the knowledge graph with the preliminary information about a business. if you go into a Business website
Mike Belson: you can get information from the website, but that's not necessarily the most accurate information because a website could say that you do a million things, And it's for SEO. But what a business really does. It becomes apparent through the work that they do in the real world. And also through the reviews that they get right and testimonials things like that. We need for our knowledge graph to be able to.
00:05:00
Mike Belson: We need for that information to be available for us to extract within the knowledge graph itself. Not from firestore the agree.
Harikrishnan V K: okay. That's true.
Mike Belson: Okay Okay, one second.
Mike Belson: Okay, and I've already shown you the chart that represents the knowledge graph, right?
Harikrishnan V K: Can you say that again? Yeah, you haven't.
Mike Belson: I've shown you a chart that represents our knowledge graph.
Harikrishnan V K: And even Bogden has also shown.
Mike Belson: Yeah, it's in Lucid chart. So, let me just open that real quick.
Harikrishnan V K: you never answer to that but yes I have since
Mike Belson: This chart over here.
Harikrishnan V K: Yes.
Mike Belson: You've seen this, right?
Mike Belson: Okay this was my concept of how the knowledge graph should be structured. and the developer that we worked with followed kind of my instructions. I actually don't know that my instructions were the right ones to follow. Okay, but I'll tell you what the idea was.
Mike Belson: And you tell me what you think, okay. so
Mike Belson: in a second.
Mike Belson: so in data in our platform we have
Mike Belson: what we call Business Services, it's like a taxonomy
Mike Belson: it classifies all the industries and then they're in every industry the categories that belong to the industry. And then the services offered Bates category and then what we call. labels and tags under each service, which represent Sort of a question and answer that can be answered after. Completing a project to help document So let's take a look real quick at home and construction. So we've done this exercise before I know but I'll just run you through this real quick home and construction And the first level this would be the industry. So this is the top level. Under home and construction. We have many categories. So this represents just two of the many.
Mike Belson: and these are Google categories. So we downloaded about 4,000 Google categories that exist and that covers pretty much all possible business categories that we have in the database, but we can add more if needed we have a UI for that. So let's take a look at under home and Construction Services roofing contractor and window contractor, And under roofing contractor, there are services. So up first level one is home construction is industry then Level 2 is Roofing Sorry is category such as Roofing and then level three. Would be the Services offered under the category so far makes sense.
00:10:00
Harikrishnan V K: cut
Mike Belson: Okay, so under services? We have like I said labels and tags. So label could be what is the problem and the answer would be the Tad missing moldy shingles torrent shingles. Okay. and you have this so this pattern this is on the
Mike Belson: this is just a sort of a pattern that can be copy pasted into every business that is in home construction and does Roofing. Now they have all these services and labels and tags already that can be copied over into their account so that they can start documenting their work, okay?
Harikrishnan V K: Okay.
Mike Belson: All right, so then over here. I'll tell you what we've done is so once you go into service. And services we have another structure. That is interesting. So for example for roof replacement, we've created something called elements. And we've kind of set this aside for the most part and we make it available for advanced users. But under elements means that a service can be in a way composed of elements and these elements are actually miles and Milestones and tasks Okay, so
Mike Belson: under Roofing on the roof replacement we have for example Milestone one, which could be
Mike Belson: since customer consultation task one is send the customer an email letting them know that. You're gonna be giving them a call task. Number two is consult with the customer. Okay, then there could be another submit proposal then it could be the actual work. Right? So we have all these milestones and tasks. the reason that we've come up with this early on was because we wanted to let our computer vision.
Mike Belson: to be objects That are correct for that are correct. And sometimes it's hard to know what objects to try to detect inside of an image. Why because you don't really have the context. But if you know the task, then you can understand the context itself. And then you can extract the right objects from the image. Does that make sense?
Harikrishnan V K: that's
Mike Belson: for example, if I'm driving down to say someone brings a car into the shop and to the car repair shop, And there's a picture of the user say bringing his car in. And the car on the background? So the question is there could be a lot of things in the background right there could be It are the trees relevant. I mean unless you know the context. then the object det might detect also trees.
Mike Belson: And trees could become kind of an important thing that it recognizes but really what's important is that we extract the things that are relevant to the task, which is currently just greet the client. So you want to take a picture of the customer more than you do of the trees, in the background or when the car is being lifted on the lift, Maybe that's when you want to take pictures or you want to detect objects that are related to I don't know say fixing the engine itself and not necessarily, the elements in the background which might be all sorts of tools that are around the shop that aren't relevant to this task itself. so what we've done is for every task we've
Mike Belson: that we run a GPT task and thousands on tasks actually and we extracted what GPD thought were relevant objects to the particular task inside of the Milestone. and then we passed that through our detection API and we use that to extract the most relevant objects within the image. The key here is to be able to we know Our customers don't want to. Do a lot. They just want to take photos. And maybe leave voice notes, but they don't want to type a lot. They don't want to do a lot of work. They just want to do is basic, the least amount of work possible Right. You can't expect someone on a roof to start typing in things into his mobile app, correct?
00:15:00
Harikrishnan V K: Good.
Mike Belson: But you can expect for him to be okay with taking photos of the work that he's doing.
Mike Belson: Makes sense so far. Okay, so
Harikrishnan V K: And contacts that we're getting is we asking them to input it also, the context that we are talking about that we are requesting the customers after capturing the photo just input that
Mike Belson: and what?
Mike Belson: Okay, so you're right. Let's say I probably don't have a screenshot of it. But before they take photos in the advanced mode, they just click on the task that they're working on. that we understand the task. We can.
Harikrishnan V K: every
Mike Belson: Figure out which objects to identify.
Harikrishnan V K: and that's been kept by open aao in here.
Mike Belson: one second
Mike Belson: Give me one second. Let me open the document itself.
Mike Belson: So this is the computer vision. This is not really it. Okay, but I opened the wrong document. one second
Harikrishnan V K: Can I get access to this one?
Mike Belson: Yeah, but give me a second. Let me find the right document.
Harikrishnan V K: Yeah, yeah.
Mike Belson: Get one.
Mike Belson: All right. You see this?
Harikrishnan V K: Yes.
Mike Belson: Okay, so I walked through the Cutting Edge object detection captioning algorithm, which has been designed to revolutionize the way we extract stories from Project documentation images. Alright algorithm is a multi-component system that seamlessly integrates object detection classification and caption generation Okay, so point is that
00:20:00
Mike Belson: you can pass into it.
Mike Belson: What am i showing you I'm showing you the wrong thing?
Mike Belson: Okay, this what I want to show you.
Mike Belson: So yeah, I was wondering why this was written with such marketing spent. So yeah, this is more of what it's about. But basically if we pass in the image urls. And I believe the labels then it will extract bounding boxes around relevant objects inside the image. And that gives us context when
Mike Belson: used in conjunction with A series of images that are taken for a particular project in other words we can take pictures. From start to finish because we know the objects and we know the task and when all these basically because we have the context and we can start weaving together our proper story with very little input from the client now. This was done several months ago. but openai now has its own computer vision, right?
Mike Belson: which is actually much better. It's crazy how good it is. So inside of our platform now. We do I don't know if I showed you but we have the ability to extract the captions from images. Have you seen that?
Harikrishnan V K: no.
Mike Belson: so if we go into our app But let me go into a demo account.
Mike Belson: So let's go into content.
Mike Belson: take a look over here.
Mike Belson: generate Auto Tags
Mike Belson: So this is I'm not sure why company image you can. the context
Mike Belson: so it doesn't have enough information here to provide the context. I think the way we have a design right now is that
Mike Belson: Okay, so this over here? If we Zoom the image.
Mike Belson: We can see it says there a bottle of soda sitting on a desk near a keyboard, right? This is using the old algorithm the one that I just showed you over here.
Harikrishnan V K: okay, okay.
Mike Belson: So this is using this And you can see it did a pretty good job, right?
Harikrishnan V K: Yeah, yeah.
Mike Belson: And we also have the bounding boxes so we know that as well. but I want to show you the new method that we're using which I think is going to be a lot more powerful.
Mike Belson: is content
Mike Belson: by the way, This is a process that should be run automatically users shouldn't have to click on anything. We want full Automation and everything. So let's try over here generate Auto Tags.
Mike Belson: Okay, so it doesn't want to give me context. Let's see if I do this.
Mike Belson: Let's try this.
00:25:00
Mike Belson: Okay, there's something that is probably. missing
Harikrishnan V K: what the numerator is rolled out to the customers. Is it only available at the demo?
Mike Belson: no, it's available, but it's I think in the prompting for this.
Mike Belson: Here, you know what?
Harikrishnan V K: Do you want to have a look at?
Mike Belson: Let's take a look. No, no, I just want you to get the concept of our system, but I can show you how this is done. So you have a better idea? I mean this probably something I can be resolved in five minutes by even because he implemented this. So the way we work is just so I come up with all the tasks pretty much. And so I have the vision and…
Harikrishnan V K: Okay.
Mike Belson: I put it into a task it out to the developers based on their skills. And they just, Implement my ideas. By the way, that's something that ultimately I'd like to hand over. because as you can imagine, there's a lot more. Going on now that we've launched our product.
Mike Belson: there's a community of hundreds of agencies. Now that depending us so we have a lot of conversations that we need to be having with all these agencies and their clients and investors there's a lot going on. So until about a few months ago, the focus was on, development now, my focus is more on, running the business so definitely opportunity to help with manage the project. But I want that's why I'm digging in deep and showing you really the kind of the fundamental thinking here around this project. And it might be going off a little bit in tangents. But I think that you have the ability to cope with this context switching.
Harikrishnan V K: Yeah, that's fine. That's not.
Mike Belson: Okay, so let me show you Rick. Let's see a computer vision since we have a problem for that. computer vision, all right, so Here's the computer vision. prompt so Over here. We should have a little expand button Okay, so act as an employee of the company who personally captured this Photograph provide a detailed description of the subject matter referenced in the context so it's missing the context notes because the context notes come through. The mobile app where we caption. What the user was talking about what the user is documenting? Have you seen our mobile app yet?
Harikrishnan V K: And not yet. You have shown me that in the video some screenshots have shown me that day. first they call you have shown.
Mike Belson: Mm- Okay, so let's do this.
Harikrishnan V K: This is something that I can install and look into it also.
Mike Belson: Yeah, Let's do that right now. So. I know you said you had to leave how much time do we have?
Harikrishnan V K: We have at least another half an hour.
Mike Belson: Okay, let's do this. and I've shown you this before this section over here all the instructions that we bring down into our users accounts originate from here, but they're concealed to them. So all they can see they can't see this. This is like a prompt that Is house only So this is used to prompt the computer vision. Why do we do it like this? Because we don't need this hidden in the code. We want to be able to optimize it. So this is a good UI for us, right? So it's using the gpt4 vision preview. and here the settings And it's house only so only I can see it. No, it's not available for
Mike Belson: For the forecasters to use they can use it, but they don't know the inner workings of it, but you can see over here the way I prompted it. So I'm telling it to think an employee of the company, who personally captured the photo and okay, so let's copy this into a text box.
Mike Belson: And all right, so I copy that now I want to show you the mobile app. Okay, so I want you to experience the mobile app actually, so I'm gonna log back in to this account over here.
00:30:00
Mike Belson: and I'm going to click on profile and then I have here an invite code. So I want to send this to you right now. and let's
Harikrishnan V K: I can do it, iPhone right?
Mike Belson: Of course.
Harikrishnan V K: Okay.
Mike Belson: iPhone or Android
Harikrishnan V K: Okay.
Mike Belson: So here you go. Let's go ahead and install it.
Harikrishnan V K: It's just downloading. Give me one minute.
Harikrishnan V K: What would be the email and password that I have to use?
Mike Belson: Let's just Put in your own email.
Harikrishnan V K: the one that I use that day, that's good.
Mike Belson: No, no, you need to create an account. Yeah. Yeah, you can do that. That's right. Put in your previous email.
Harikrishnan V K: Yeah, I'm blocked.
Mike Belson: Okay, cool.
Harikrishnan V K: So let's get our first project.
Mike Belson: that's right. three seven six seven
Mike Belson: and I'm going to share my screen too.
Harikrishnan V K: I have to upload a photo. Okay?
Harikrishnan V K: Do you want me to proceed with that or just wait?
Mike Belson: Yeah, go ahead good just pretend like you're a user.
Harikrishnan V K: Okay.
Mike Belson: mmm
Harikrishnan V K: okay, there seems to be some promotional issue once and
Mike Belson: Set a problem.
Harikrishnan V K: no permission issue. It says that I gave it limited access to photos, but it's asking full access.
Mike Belson: yeah, I need to give it full access.
Mike Belson: So let me do it in my end here real quick also, okay, so you can see my screen now,…
Harikrishnan V K: Yeah. Yeah.
Mike Belson: Okay, so I'm gonna click plus I'm just gonna give it a address. Let's say in Los Angeles somewhere. and I'll say
00:35:00
Mike Belson: In this phase. We are doing demolition of the home the clear space for an expanded click next kitchen remodel Can't okay.
Harikrishnan V K: Yeah.
Mike Belson: So this is more of the advanced version. I think yeah where you have here also milestones and tasks so
Mike Belson: open the project. And over here. I can click and update your tasks and you see all these tasks over here.
Harikrishnan V K: 
Mike Belson: So let's say prepare installation site and we click over here and start. Plus. I'll say let's take a picture of this. and the picture of this
Mike Belson: Okay, click them done
Harikrishnan V K: this is the installation.
Mike Belson: Yeah, this is just an example of a picture that I found online. and now I can So now click on
Mike Belson: on the status I could say This stage is done and I can go back over here. And now I can continue so it sees that I skipped certain parts. Also, there's a duplicates over here, which is a problem that I think we already resolved. actually
Mike Belson: No, this isn't I'm saying that I thought it was resolved. Maybe they didn't deploy to production but just having duplicate tasks over here is obviously Okay, actually. Yeah, there are duplicates here prepare for consultation twice. And conduct consultation is twice. So there's duplicates over here. So yeah, that's
Harikrishnan V K: You had no doubt. Yeah, that's a consistent duplicate.
Mike Belson: Yeah, so again duplicates over here. Definitely not good. if I click over here on this photo I can say add information. Okay, so that's the context that the prompt is actually using so let's click on.
Harikrishnan V K: Yeah.
Mike Belson: This is the area where we're looking to expand the kitchen into So currently we're doing demolition. All…
Harikrishnan V K: he
Mike Belson: so let's do that. Let's click on done. and we have this activity feed okay where you can see Your photos and you can see also other people's photos from the team. So whoever's taking photos. You can see the work that they're doing. So this is kind of like the collaborative aspect of it. And in most settings there't The tasks is something that is for advanced users. Because we also need to set up tasks in the back end. And that match up.
Harikrishnan V K: thing
Mike Belson: That particular business's needs. All right. I don't want to get ahead of myself with that, but I did want to
Mike Belson: okay, let me
Mike Belson: let me switch back to the desktop.
Mike Belson: Let's go into our content. So it'd be interesting if the computer vision is having trouble because it's a screenshot. It's kind of pixelized.
Mike Belson: Let's So now it has the context in the description and this should work.
Mike Belson: So now it wrote up something that is a lot more complete. No, it's hard to read it here. So I'll put it down here below.
Harikrishnan V K: And this is from the open am model that we talked about.
Mike Belson: Yeah, this is from The Prompt that I showed you earlier. So in this Photograph we're looking at an interior space currently undergoing demolition.
Harikrishnan V K: Yeah.
Mike Belson: It's part of the renovation project. So you understand it. Why is it no this because I gave it context and my voice note, right?
Mike Belson: So let's take a look.
Harikrishnan V K: but
Mike Belson: Let's take a look at the prompt. This was the prompt. I'm going to start over here. This was the prompt. If we go into the project itself, this was my description over here. So let's go over here. and we'll say context note And then the context node gets replaced with this placeholder. And it gets replaced with that and then there's also project title. which is Gonna be this over here. oops
00:40:00
Mike Belson: So now it has much better context. if we go into a Open AI playground. And we run this.
Mike Belson: Then here let me switch my screen.
Mike Belson: So let's copy this now.
Mike Belson: And let's run this.
Harikrishnan V K: presentation you
Harikrishnan V K: but I have one query. So right it's not actually analyzing the photo.
Mike Belson: You have what?
Mike Belson: Okay, let's take a look here. The photographic captured showcases the initial phase of our kitchen Expansion Project where the demolition is currently underway in the image you can see the remnants of what? Was once a partition wall. with exposed wooden studs standing there where drywall has been removed. This is crazy. Okay, this is really insightful. this
Harikrishnan V K: yes, no. No, it's being said I know I just wanted to query was like because you might have taken a photo of let's say a certain portion of the demolition
Mike Belson: let's take a look at the photo. is let's say I'll share with you the photo.
Harikrishnan V K: because the output is based on what is given as a context note.
Mike Belson: So you can see it. This is the photo that I took and…
Harikrishnan V K: Okay.
Mike Belson: it was able to see things that are just Okay. so
Mike Belson: the pile of the breeze and the floor consists of broken pieces of what's that?
Harikrishnan V K: Nobody you do not have the photo you. Did not add the photo the photo was not the playground.
Mike Belson: Of course I did. It …
Harikrishnan V K: the playground
Mike Belson: hold on. No, that's the thing. let me walk you through the process again, okay.
Mike Belson: Stop presenting I'm going to take you into.
Mike Belson: Stick a look at the mobile app. Let me share my entire screen actually.
Mike Belson: Can you see my screen or are you working on a small laptop?
Harikrishnan V K: No, yeah, I can see that.
Mike Belson: Okay, in the mobile app this photo over here gets this caption.
Harikrishnan V K: Yeah.
Mike Belson: This is my voice note over here.
Harikrishnan V K: Yeah, based on the voice that you have added that.
Mike Belson: The top is my voice note, correct? Okay,…
Harikrishnan V K: Yes.
Mike Belson: The app over here. Let's go into this image, And this is the voice note in the description. I have the voice node. I run the on The computer vision prompt I run it.
Harikrishnan V K: Who?
Mike Belson: Let's take a look. I run it like this.
Mike Belson: This is what it gets it gets the image and it gets the context note and it gets this should actually the project title. And it says over here. As the employee of our company's who personally capture this photo provide a detailed description of the subject matter referenced in the context So this is the area where we're looking to expand the kitchen. Into So currently we're doing demolition. Below and feature and featured in the image. The photograph looks like the information in the context not describe the appearance of the relevant subject noting the specifics of their state action condition placement. So all these things so this is part of the caption. Sorry. this is the prompt that
00:45:00
Mike Belson: is used in conjunction with the context node and the project title. So the result is now that open AI has this prompt with the image. It says in this photo we're looking at interior space. Currently undergoing demolition is part of a home renovation project. This is what we read before the room is in a state of disarray and we can see here. It's actually in a serious state of disarray right Here we can see this was the image over here? Okay.
Harikrishnan V K: yeah, yeah.
Mike Belson: All right, so we can see the room is actually in the state of disarray. Let's see what else it picks up. So this is the image. This is In particular, so just to make sure that's All right. The room is in the state of disarray indicative the debt of the demolition phase with debris scattered across the floor, The debris consists of large chunks of drywall. This is unbelievable Insight it knows that this was from here. pieces of wood framing and other construction materials that have been removed from the wall from the walls and ceiling okay. The floor is covered with and plaster and dust
Mike Belson: and we can That we see over here exposed studs.
Harikrishnan V K: Yeah. Yeah, I know this match. No only way I had was lucky. Okay here you have called The Vision API and you have got that output which is actually matching but when you're done So here you use the openas Vision one to get that is that right?
Mike Belson: One might Hold on. Say that again Harry.
Mike Belson: right Right. Yeah.
Harikrishnan V K: Yeah, but the playground one was not that right playground is not that.
Mike Belson: in the playground, let's take a look at the playground one second.
Mike Belson: where to go over here The playground is giving me. The room it's very similar. I just ran the same prompt. I Okay,…
Mike Belson: I understand what you're saying. You're right.
Harikrishnan V K: Yeah, but we don't…
Harikrishnan V K: but you go to the photo.
Mike Belson: You're right. You're right. Yeah. Yeah. No, you're right. You're right. My apologies. You're right a hundred percent. So this needs to be done in conjunction with the image, So this wouldn't work individual independently, you're right. But it was just hallucinating. It was imagining some photo. Okay, very good.
Harikrishnan V K: Yes.
Mike Belson: So you're right. there's no vision.
Mike Belson: prompt over here So if there was I'd be up I'd upload also the computer, I could do in.
Harikrishnan V K: Yeah, we could have had.
Mike Belson: In chat GPT, but this is the result so why is this really important? where do I want to take things? Number one? This is really really important. This is very valuable because now we can use this information and We can use it to tell a story. so if we have pictures of all the phases of the project we can know in very great detail. A lot of what happened inside the photo, correct and inside the project without users having to tell us a whole lot true.
Harikrishnan V K: Correct.
Mike Belson: There's another aspect to this I want to
Mike Belson: put this into our knowledge graph, okay.
Harikrishnan V K: 
Mike Belson: I want inner Knowledge Graph. Let's go back to Cipher.
Mike Belson: I want inner Knowledge Graph to have this is the project.
Mike Belson: So a project is a Content? And has a category. So this is a content of hype project in our system, but has a media. And this media has all sorts of objects related to let's say there's a better example.
00:50:00
Mike Belson: So in this case, here's a The Milestone has a task and the task has all these media. and these media have these objects related to them. So here for example.
Harikrishnan V K: again
Mike Belson: is an object that
Mike Belson: Has a bounding box for example. Has an embedding. Okay, so this could be over here. For example, this could have been. Drywall there's drywall in the image or there's wood framing in the image things like that, right?
Mike Belson: So Okay,
Harikrishnan V K: Okay.
Mike Belson: I want to populate the draft with all this information about every image so that we can have now when we're telling us a story about this we can extract the knowledge and be able to tell a proper story. The images should be in the right chronological order. relating to the tasks that took place and into those have embeddings and every object detected within every image so that if for example we're talking about.
Mike Belson: an object that is drywall and during a demolition process and we want to engage, to provide the user with other examples of drywall work that we've done. Then we can instantly do a search in the knowledge graph and find other projects we've worked on that had similar situations. when we create content we can say if you're looking, here's another three projects that we've done that. In which we've demolished. kitchens or living rooms that had drywall. things like that. So it's very important for me to be able to extract a lot of information, as much information from the system as possible. This makes sense.
Harikrishnan V K: Yes that does.
Mike Belson: Okay How do these two so how does this all come together over here? We could easily run a prompt let's see over here this is the story. So I could say over here.
Mike Belson: Something like this. Please list. all of the objects described here. and order by priority according to the
Mike Belson: contest of
Mike Belson: Okay, let's probably a bad prompt but let's do that.
Mike Belson: Debris exposed that electrical wires. Okay, this is really cool. So these are the most important things in this image, so I could update the prompt and…
Harikrishnan V K: he
Mike Belson: I could just tell it to just give me these things without everything in the
Mike Belson: inside these just it could give me just debris exposed studs electrical wires. And then what I could do is I could run Through this API over here to get me the bounding boxes, too. Right. So now I could actually have visibility Windows debris like bounding boxes. And if I wanted I could show examples of other. projects that had debris and specifically zoom in or just show the debris parts of it. So using this API in conjunction could give me the bounding boxes of the most important elements within the image itself, correct?
Harikrishnan V K: that's Yeah, and you can reverse look up it when someone is asking for it in a chat.
Mike Belson: exactly. Okay this so now we're in a position to really make magic happen with all these images that people are taking. And we can interchat bot have really good conversations with people. I mean a person pay. can you show me some other work were electrical outlets were exposed? Okay fine, and it would know how to do that. It would be able to fetch that information from the chat, but that would be
00:55:00
Mike Belson: in a sale in sales mode the chat itself would know. This user is a little skeptical about our capabilities with drywall he's asking all these questions about drywall he thinks maybe we haven't done drywall because he thinks that maybe we just do for example brick type of homes, but we do also drywall homes or office spaces and let me show him, some drywall work that we've done now that the sales agent can become super intelligent super capable, So by extracting this information, but by Available within our chatbot experience we can offer businesses. Much more engaging conversations that lead to much higher conversions. make sense
Harikrishnan V K: Yes.
Mike Belson: Okay.
Mike Belson: my concept for the knowledge graph being that I'm not a knowledge grass expert. And this has been months ago. and we kind of designed the knowledge graph around this concept was that and the one hand we have all the graph of all the industries category services. labels and tags on the one hand just so that we can apply them to each business and in a way it gives us a Sort of a schema. I would say that can be applied Maybe
Mike Belson: I think the idea here was as we're working with multiple different businesses and as their populating, their nodes are being populated we can also update or create relationships to this separate structure over here so that we can have don't learn, just generate domain expertise. So, I don't know that that's the right approach though to be honest.
Mike Belson: In other words, if we have here a kitchen remodel there could be a link to a kitchen remodel over here. And this is on the business level. but in the house level we would say Okay, so this kitchen remodel note over here is related to 2000 businesses over here that do kitchen remodels, or that
Harikrishnan V K: within the same agency or
Mike Belson: 2000 no, this is no so this is house. and then Yeah the house account and…
Harikrishnan V K: okay.
Mike Belson: then the house account, has this. Relationship to all the businesses that are underneath the agencies. I think there's probably more efficient ways to do this.
Mike Belson: I don't know that we need to have our own mapping of or this is kind of maybe A summarization of what's going on in the knowledge graph itself, but I don't know if that's needed. What do you think?
Harikrishnan V K: No, so that because I will spend some more time in understanding that Knowledge I will just logged into it. I can answer that book and give me so I'm able to see the Gap now. I will just spend some time in going over it before commenting.
Mike Belson: Okay, that's fine. and…
Harikrishnan V K: Okay, because
Mike Belson: then the other concept that I had was. and this was really important for me is like I said initially we learn about the business from scanning their web presence. So that gives us initial start a good starting point, but then as they start documenting their work. this knowledge graph starts growing and growing and growing right with all these projects. So we have all these projects. Related to the business, but then we have also here a map. So I called the business level taxonomy. So here we have a representation of all the services. that they're completing all the projects that they're completing in the real world and
01:00:00
Mike Belson: the more for example
Mike Belson: let's take a look here. So for example tour say torn shingles, which is a tag under.
Mike Belson: under problem related to roof repairs have problems such as throwing shingles. And a torn shingle has this embedding over here. as an example. So if there's a torn shing on this and in a project and this over here. It has a relationship to that embedding meaning that. This over here becomes sort of.
Mike Belson: Amplified in terms of its importance if there's a lot of Projects that are tagged with torn shingle then obviously this becomes an important kind of a tag. Okay, and this over here moldy shingles is an even more important tag what this does is it allows for our sales chatbot to gravitate towards what the businesses core competencies are. so for example if there's a tag such as
Mike Belson: I don't know
Mike Belson: Maybe something a problem related to wind.
Harikrishnan V K: You to wait.
Mike Belson: And that there aren't a lot of projects that are tagged that way then the AI when it's trying to sell the user isn't recommending or isn't suggesting. Is it related to wind damage? It's going to ask. First of all, is it related to moldy shingles? And because this business does a lot of moldy shingle repairs most likely the user is going to say yes. But then it's going to offer this option or maybe it offers a number of options to the user because the way our chatbot works is it prompts the user with options that are most likely okay, just to make the conversation very simple instead of asking open-ended questions all the time. So it kind of prompts the user so that the user doesn't need to think too much.
Mike Belson: Which in a way makes the conversation a little smoother…
Harikrishnan V K: 
Mike Belson: because people sometimes don't know exactly They don't know what terms to use. They don't know how to explain themselves when it comes to these kind of things. But if in the conversation we could say something like okay, it's related to moldy shingles and also show them they might not know what model shingles are or torn shingles are and then if in the conversation we could show so examples of moldy shingles and torch angles because we already have the projects and we have the photos now, it becomes super fun experience. You get it?
Harikrishnan V K: yeah.
Mike Belson: Does it make sense?
Harikrishnan V K: Yeah.
Harikrishnan V K: Yeah that does.
Mike Belson: but if we have wind, when the Damaged shingles, which aren't important because it's a small tag. They want there aren't a lot of projects tagged that way then it's not going to show that. it may be the customer then says, what for us. It's wind related damage. so the chatbots still knows that they do this kind of work and it can still show those kind of examples and get the customer to do business with them based on examples, but it's not a big deal, so this is how we learn about businesses. So we see over here this business does mostly roof repair. They also do
Mike Belson: The also potentially could do other services but this is the most meaningful one so it knows how to steer the user. Towards the right outcome. this was my Kind of thinking and how it should be done. But maybe there's other algorithms that don't require the side of the knowledge draft in order to steer the user in the right direction or in order to have these, instant understanding about a business. This is almost like a summary of the business metrics maybe or of this what are your metrics when it comes to multi shingles? Yeah, we've done 35 projects in the past 30 days. Okay, how about torn shingles 22 projects? Okay, so this is sort of
01:05:00
Mike Belson: a summary of what the business is all about. And this way we can drive the users and in the right direction through conversations and through content generation. But my idea was that if we just have a Knowledge Graph like this I wouldn't really know. As an unexpert how to use all this knowledge in real time to have conversations about what's important and maybe I'm not doing it right. So maybe the knowledge graph can in real time. Just give you instant. The stats and what the company is all about?
Mike Belson: What do you think?
Harikrishnan V K: Yeah, yeah this mind map also. Can I get it assistant? this mind map
Mike Belson: This what? Yeah.
Mike Belson: Let me know if you can open it.
Mike Belson: What's your email?
Mike Belson: Yeah, one second. Let me
Mike Belson: I'll put it in slap. It's fine. Pick it up for me.
Mike Belson: It's so check your email.
Mike Belson: Okay, I put it in Slack.
Mike Belson: All right, ask Bogdan for access to everything whatever you need.
Harikrishnan V K: not yet.
Harikrishnan V K: Okay.
Mike Belson: I would there's probably plenty of discrepancies there. but
Harikrishnan V K: Yet a neo4j I just logged in that bug…
Mike Belson: this is
Harikrishnan V K: then I have what I will do is that I will just cost so if I do that with the map that you're shared with me.
Mike Belson: The map is a representation of my idea of…
Mike Belson: how things should be done. And I'm not an expert Knowledge Graph guy.
Harikrishnan V K: Okay, and…
Harikrishnan V K: I will just spend time.
Mike Belson: Okay, so this is just my view of how the world should work,…
Harikrishnan V K: But mapping one that is towards our goal, right?
Mike Belson: but I might be wrong. so there might be capabilities with Built-in capabilities of neo4j that don't require for us to create this side of the knowledge graph to create this part over here.
Harikrishnan V K: Okay.
Mike Belson: that's really why I want to be able to tap into your expertise And get your idea of what's really important, but on the other hand the value over here. So is that we can create embeddings? For each of these items over here and then using the embeddings. if something over here is called for example torn shingle, but in here it's classified as a shingles that was torn. It will still be the same thing and there will still be a relationship between them and this. Over here, we'll just growth, due to the nature of more. relationships between the entities on this side and then that side and again.
Mike Belson: Maybe there's other ways of doing it. So maybe just anyway, this is
Mike Belson: okay.
Harikrishnan V K: Differently. I'll just check about it. So what I will do is I will just go over you can send me this recording link also,…
01:10:00
Mike Belson: Okay, sounds good.
Harikrishnan V K: so that might help me and…
Mike Belson: Let's do that.
Harikrishnan V K: I will spend time in this map and then you focus and maybe Tuesday we can talk on what's the best approach and…
Mike Belson: Okay, sounds good.
Harikrishnan V K: how to take it further?
Mike Belson: Alright, I'll talk to you later.
Harikrishnan V K: Yeah, so today maybe I might not be able to spend on it,…
Mike Belson: Okay, sounds good. Okay. Bye.
Harikrishnan V K: but I'm planning to spend time on this on Saturday and Monday.
Mike Belson: Have a nice weekend. Bye.
Harikrishnan V K: And please do Shadow accordingly so credit I can definitely have a little bit. Okay, okay. Yeah, thanks, Mike.
Meeting ended after 01:10:50 👋
mikeb1 commented 9 months ago

Meeting Outline: Discussion on Technical Issues and Knowledge Graph Strategy

Date and Time

Attendees

Topics of Discussion

Technical Issues with Bot and Website Loading

Knowledge Graph and Data Management

Computer Vision and Image Analysis

Sales Chatbot and Customer Engagement

Strategy and Project Management

Action Items

Next Steps

Meeting Conclusion

mikeb1 commented 9 months ago 
Act as a Neo4J expert with 15 years of experience and make any recommendations
mikeb1 commented 9 months ago

As a Neo4J expert with 15 years of experience, I would make the following recommendations based on the conversation outlined above:

Knowledge Graph Structure and Optimization:

  1. Graph Model Review:

    • Conduct a thorough review of the current knowledge graph structure to ensure it aligns with the business objectives and data relationships.
    • Simplify the graph model where possible to reduce complexity and improve performance.

  2. Schema Design:

    • Implement a flexible schema that can accommodate changes in business services and categories.
    • Use labels, relationships, and property keys effectively to represent the taxonomy of services and their attributes.

  3. Data Ingestion and Integrity:

    • Establish robust procedures for data ingestion from Firestore to the knowledge graph, ensuring data integrity and consistency.
    • Implement data validation rules to prevent incorrect or duplicate data entries.

  4. Real-time Data Processing:

    • Explore the use of Neo4J's transactional capabilities to process data in real-time, which can enhance the chatbot's responsiveness and accuracy.

  5. Graph Algorithms:

    • Utilize built-in graph algorithms for insights, such as centrality measures to identify key services or categories, and community detection for grouping similar business services.

  6. Embeddings and Machine Learning:

    • Integrate graph embeddings and machine learning models to capture the nuances of business services and improve the chatbot's recommendation system.
    • Consider using Neo4J's Graph Data Science library to build these models.

Computer Vision Integration:

  1. Image Data Linkage:

    • Ensure that image data and the results of computer vision analysis are linked to relevant nodes and relationships within the knowledge graph.
    • Use graph relationships to connect tasks, milestones, and media for a comprehensive understanding of project documentation.

  2. Contextual Analysis:

    • Enhance the chatbot's capabilities by incorporating image context into the knowledge graph, allowing for more nuanced conversations based on visual project data.

Chatbot and Customer Engagement:

  1. Dynamic Conversation Paths:

    • Develop dynamic conversation paths in the chatbot based on the knowledge graph's insights, allowing for personalized and relevant interactions with users.

  2. Content Generation:

    • Use the knowledge graph to inform content generation, creating narratives that align with the business's expertise and customer interests.

  3. Feedback Loop:

    • Implement a feedback loop where chatbot interactions can inform and update the knowledge graph, leading to continuous improvement in customer engagement.

Project Management and Collaboration:

  1. Access Control and Permissions:

    • Set up fine-grained access control within Neo4J to manage who can view or edit different parts of the graph, ensuring data security and integrity.

  2. Collaborative Development:

    • Foster a collaborative environment where developers, data scientists, and business stakeholders can contribute to the evolution of the knowledge graph.

  3. Monitoring and Maintenance:

    • Regularly monitor the performance and health of the knowledge graph, and perform maintenance tasks such as index optimization and refactoring of nodes and relationships.

Training and Documentation:

  1. Knowledge Transfer:

    • Provide comprehensive training to the team on Neo4J's capabilities and best practices for maintaining and evolving the knowledge graph.

  2. Documentation:

    • Create detailed documentation of the knowledge graph's structure, data models, and integration points to facilitate understanding and future development.

By implementing these recommendations, the knowledge graph can become a powerful asset for the business, driving intelligent chatbot interactions, enhancing customer engagement, and providing valuable insights into business operations and services.