UI for FashionAnalysis

[SreenivasDega]

To develop user interface using HTML to publish all the data crawled from fashion sites, similar to Myntra layout.

[MohanaVedingadu]

please find below update,

1. Implented to UI for populating all product but I am facing complexity to add date picker in nav bar.
2. I am able to get the data in database based on the dates range and filters and sources, implemented apis
3. Need to work on filters in frant end and intigration part

[MohanaVedingadu]

please find below updates,

1. Added filters for dates, sort by, source are completed
2. But each time by selecting any filter reload entire page along with the headers part(filters) and reload all the filters which are present in before the request
3. We need fix reload problem for every request only items body reload not for filters

[MohanaVedingadu]

1. I tried by changing the request format of api, but its not working means items are not updating.
2. After adding the filters in the URL then only items are changed but these complete page is reloaded.
3. I tried with Ajax calls, Date picker not initialise. please suggest me on this.

[MohanaVedingadu]

1.I fixed reload issue. 2.I added ratting and icons for each item. 3.In database duplicate items are present. 4.SQL Query takes too much of time, please suggest me on this. 5.Need to add the pagination for the UI Page.

In database myntra and nnow only products are available.

[MohanaVedingadu]

[MohanaVedingadu]

Please find below updates,

1. In first half I worked on MYSQL query by using index along with jaffrin, I tried many ways but its takes too much of time by fetching the data because of duplications Total Number of records -- 6491949 Total number of distinct hd_id records -- 863293 we need to extract the data from these many records so its taking too much of time for getting the data.
2. I have worked amazon and ajio data is not populating in site, these issue is fixed, as of now there is no records for amazon. I added icons for all these sources.

[MohanaVedingadu]

[MohanaVedingadu]

Please find below updates,

1. Fixed date picker issue
2. Added extra filters for sortBy a.What's New b.Better Discount c.Price: Low to High d.Price:High to Low
3. Added product detail link for product its working now.
4. I worked on pagination but its not completed need to work on pagination

Note: Need to work on

1. Pagination 2.Grouping the item using productId 3.product url is wrong for ajio site
2. Some of UI changes as discussing yesterday
3. Need to add the rating points in database and change spider accordingly.

[MohanaVedingadu]

+---------------------------------------------------------------------------------------+ | reference_url | +---------------------------------------------------------------------------------------+ | https://www.ajio.com/steve-madden-genuine-leather-shimmery-ballerinas/p/410006469_387 | | https://www.ajio.com/steve-madden-genuine-leather-metallic-ballerinas/p/410006469_18 | | https://www.ajio.com/steve-madden-genuine-leather-ballerinas/p/410006469_17 | | https://www.ajio.com/s/dc-shoes-3483-6296 | | https://www.ajio.com/steve-madden-slip-on-driver-penny-loafers/p/410012164_217 | | https://www.ajio.com/steve-madden-slip-on-driver-penny-loafers/p/410012164_417 | | https://www.ajio.com/superdry-commodity-skinny-chinos/p/410012634_04p | | https://www.ajio.com/superdry-standard-low-rise-skinny-jeans/p/410012951_fwe | | https://www.ajio.com/superdry-striped-sleeveless-fit--flare-dress/p/410015889_phj | | https://www.ajio.com/superdry-strappy-tie--dye-playsuit/p/410015894_52g | | https://www.ajio.com/superdry-super-swimsuit-with-tie-up/p/410015923_02a | | https://www.ajio.com/superdry-double-striped-v-neck-top/p/410015936_11s | | https://www.ajio.com/superdry-floral-print-fit--flare-dress/p/410017337_34r | | https://www.ajio.com/gas-mid-rise-slim-fit-whiskered-jeans/p/410020121_wn39 | | https://www.ajio.com/gas-ralph-slim-fit-polo-t-shirt/p/410020257_1525 | | https://www.ajio.com/s/gas-3483-6296 | | https://www.ajio.com/gas-ralph-s-3-slim-fit-polo-t-shirt/p/410020257_0699 | | https://www.ajio.com/gas-anders-k-lightly-washed-slim-fit-jeans/p/410021674_wk22 | | https://www.ajio.com/gas-mid-washed-straight-fit-norton-jeans/p/410021817_wk79 | | https://www.ajio.com/superdry-mid-rise-skinny-fit-jeans/p/410022144_foq | +---------------------------------------------------------------------------------------+

[MohanaVedingadu]

[MohanaVedingadu]

[MohanaVedingadu]

Please find below update,

1. Worked on SQL query but its takes too much of time
2. Worked date range picker, its not populating single date(today,yesterday)
3. Worked on ajio product url, but we are crawling two different url so reference url is not matching for some times need to work on that
4. Checked category and subcategory for nnow and ajio site needs to work on because in ajio site main category also not storing and crawled all the url we need to find the category.

As discuss yesterday evening stand up we need to change all schema and all queries.

[MohanaVedingadu]

I changed the spiders according to schema changed all the queries according to schema Myntra and nnow is completed, Now I am working on ajio Myntra and Nnnow issues:

1. availablity populating wrongly in myntra
2. rating points added both myntra and nnow
3. category and sub category are added for nnow
4. changed return item regarding to schema Now I am working on ajio issues: 1.reference url is not matching for some time 2.size populating wrongly,like size: ['FS']
5. need to add category and sub category

[MohanaVedingadu]

[MohanaVedingadu]

please find the below updates,

1. Added site logo over the product image
2. Added product detail link for product card so that we are click on any position on product its redirecting to product detail page.
3. Added search box for getting brand name, added new api for getting brands in database Need to work on integration part, I getting complexity on javascript.

Note:By entering the letter we are searching the item from the database its takes too much of time for getting data, by open the site firstly we are getting the all brands which are present in the database and maintain that list in client side and by enter the key we are extracting the brands from the list.

[MohanaVedingadu]

1. Added source filter dynamically
2. Working on queries optimisation but ORDER BY take time by using mysql query, so yesterday discuss with charan today I will try with python sort by
3. Added brands filter
4. fixed mrp and selleing price zero for all the product issue for ajio site

[MohanaVedingadu]

please find below updates,

1. Worked with farooq to fix pagination issue
2. Added loading page
3. fixed filter not working for combination of sort By low to high and source Myntra
4. I tried to change the queries by using python but I got Memory error, so I cant able to proceed next step so please suggest me on this

[MohanaVedingadu]

@SreenivasDega , @malla794 Please suggest me on this

cur.execute("select brand,reference_url,ratings_count,category,discount_percentage,source,selling_price,mrp,image_url,title from products_info where DATE(created_at)>=%s and DATE(created_at)<= %s and source=%s and category=%s and brand=%s ORDER BY discount_percentage DESC limit %s offset %s;",[start_date, end_date, source, category, brandName,limit, offset])

This query is used for getting the data according to filters, but its takes to much of time to extract the data.

[MohanaVedingadu]

[MohanaVedingadu]

[MohanaVedingadu]

1. I tried to get the data in mysql using Djnago ORM but there is no time difference between the Flask and Django, This issue because of MYSQL Query takes more time.
2. I tried to dump the data from MYSQL to MongoDB, because MongoDB is faster than MYSQL but I cant able to dump the data please suggest me how to solve this issue.

[MohanaVedingadu]

please find below updates, 1.I changed all the queries to read the data from MYSQL using pandas

2. Added search brand filter queries and its working now All changes are in production, please check

[MohanaVedingadu]

[MohanaVedingadu]

please find below updates, 1.added all category options for category filters Ex: Men,Women,Boys,Girls,Unisex,Unisex Kids 2.Fixed search bar issue , changed all the queries and frontend changes

3. Added date filter for search brands
4. fixed ajio and myntra price zero issue
5. verified better discount shows products which are selling price is greater than mrp price, because of ajio records are not updating, I need to check in spider.

[MohanaVedingadu]

1.fixed issues in ajio site. a.I have worked on ajio prices issue and data type issue. b.fixed data loading issue for ajio site. c.fixed is_availability empty issue in ajio, now we are populating based on the price details availability.

2. working on US data populating in fashion site,I have done all the backend apis, now working on front end part.

[MohanaVedingadu]

Please find below updates,

1. changed all the queries so that data is loaded speadly.
2. changed search api to different api, now first load also speed only. Reference link: http://116.203.124.171:5002/
3. Added US data in fasion site all backend apis and integration done, front end design is pending. because no space to add country filter in nav bar so that planed to add country flag in navbar

[MohanaVedingadu]

[MohanaVedingadu]

Added US data in fashion site, handled all the filters and pagination according to country filter, by adding country flag some java script issues occurred because bootstrap version are not matching, so I need to work on low to High filter and by selecting myntra its takes too much time.

[MohanaVedingadu]

Please find below updates,

1. I tried to debug the loading issue there is no api issues, sql query takes time to load, @malla794 please help me on this
2. I have finalize the all the unique sub categories from all the sources, need to add mapping for all the fashion spiders.
3. I have tried to debug the myntra data not updating issue, luminati proxy not working in production, so myntra crawling is not working.