search

wet-boew / wet-boew-drupal

Drupal variant of the Web Experience Toolkit (WET)
137 stars 74 forks source link

Proactive Disclosure #115

Closed joejoseph00 closed 10 years ago

joejoseph00 commented 12 years ago

in response to @sylus , currently I've completed a barebones content types for PD (proactive disclosure)

Done: PD Contract

Todo: (I'm working on these, hopefully done by end of day tomorrow) PD Hospitality PD Travel PD Reclassification

MORE Todo:

The plan is to export this to a feature for collaboration and versioning and then eventually turn it into an app if needed.

Challenge #1: the translations do not export to the feature when I tested it, so to mitigate this I'm putting english and french in the description fields so that the translations can be more easily done at the end of the process.

I've never used Github before but I'll try to put the feature up here somewhere by thursday or friday. It will have the basic content types and fields.

I've got the luxury of having a Drupal 6 implementation to look at but have not been able to duplicate the money field because the money field module doesn't exist in Drupal 7 so for now it's just a text field (oh well, format it with code I guess, put a dollar sign there, validate)

Dependancies: http://ftp.drupal.org/files/projects/references-7.x-2.0.tar.gz node reference, not exactly sure why yet this is needed, but it's in our Drupal 6 implementation

workflow: I will try to document in detail the workflow requirements by next week or the week after.

Note: I'm doing this as a Drupal 7 learning exercise, so far I've learned quite a bit.

joejoseph00 commented 12 years ago

I created a new feature (140kb tar file) based on my content types and I got this error when trying to install it on another server: FieldException: Cannot change an existing field's type. in field_update_field() (line 234 of /srv/www/htdocs/wet.7pro.ca/modules/field/field.crud.inc). Drupal refuses to disable this feature (module) now. So to save time I'll probably just blast this instance and get a new build. Then I'll try it again, making smaller features.

Perhaps because I'm using entity reference , so, my next plan is to create smaller features, this time start with the entity that is the dependency.

joejoseph00 commented 12 years ago

Ok, I reloaded my test environment with a fresh db and the feature loaded properly. If you want to download it , get it here (for now) http://wet.7pro.ca/sites/wet.7pro.ca/files/pd-7.x-1.17-alpha.tar

sylus commented 12 years ago

Thanks @joejoseph00 for taking the time I will take a look at this this week! :) :)

joejoseph00 commented 12 years ago

The feature includes 4 content types, with various fields, it's missing a few things from the source. I noticed that the auto node titles I had in the source did not make it into the destination when I installed the feature into my test server #2 ( http://wet.7pro.ca ), even when I installed all the dependancies before installing the PD feature some of the settings were lost during feature creation or somewhere, not sure yet. Everything else went in except for that and the translations I had written. I will change the datatype of the money fields from text to decimal or money depending on how good the new D7 money field is (burried somewhere in drupal.org)

The real work is in upgrading our D6 PD module. Not sure how I will integrate this with the feature or if I will just configure our future PD "module" as a dependancy of the PD "Feature". Both will be required for this to work. Or combine them.

I'm going to run coder upgrader on our D6 pd module today, then I could in theory put it either on my personal SVN server (http://usvn.7pro.ca ) grant read access to anyone (username: wet password: wet ) and r/w for those who ask.

I'll keep you guys updated on this, I got some timely advice from a friendly bird so now it's time for work

joejoseph00 commented 12 years ago

I isolated the PD code from our previous version on D6. Then I ran coder upgrader to do the upgrade on friday, but I haven't done the manual upgrade tasks yet, I haven't yet tested what came out of coder upgrader but it's comming. There's not a whole lot of code, but the work will be to figure out how it integrates with the rest of the PD components (content types, views, etc).

joejoseph00 commented 12 years ago

Making some headway, but still lots of work to do so far here's a list of dependancies:

views views FILE_ELEMENT Feature called pd (pd content types) this feature is to be included with this module Workflow - http://ftp.drupal.org/files/projects/workflow-7.x-1.0.tar.gz

automatic nodetitles Job Scheduler Feeds Formatter php filter computed field - http://ftp.drupal.org/files/projects/computed_field-7.x-1.0-beta1.tar.gz

sylus commented 12 years ago

Really awesome work going into this :) Super excited!

sylus commented 12 years ago

Some problems with permissions on Team created an issue for it at: #172

joejoseph00 commented 12 years ago

Status update, items completed: 1) Isolated PD code from D6 CTA module 2) Renamed CTA module to pdwet 3) Ran coder upgrader, reviewed code that was upgraded to D7 4) added dependancies to new D7 module 5) resolved a few minor bugs related to the upgrade 6) created the content types, fields, etc, and content type settings, using money fields, one computed field 7) created a worklow , but actions not working yet (see notes below) 8) created a feature that contains all content types, settings , workflows, to date
here is the feature tar file: http://wet.7pro.ca/sites/wet.7pro.ca/files/pd-7.x-1.17-alpha.tar here is the README file http://wet.7pro.ca/sites/wet.7pro.ca/files/README.txt

Workflow actions in D7 gave me problems, tried Workflow Rules and it crashed.

I installed the workflow module into D7, reproduced the two workflows for PD (Other PD) and (Travel/Hospitality) as it is used in our D6 implementation with the states: Draft Comms Verified Published

Then tried to create the action to send an e-mail when the state changes from Draft to Comms (as is in the D6 version)

This didn't work when I tried it, I found a discussion about missing triggers but I don't know if it is related to my issue, I'll have to test this to find out.

See the discussion here: http://drupal.org/node/1380954

and the patch (I will try to test this today. not sure if it will solve my problem) http://drupal.org/files/triggers-1380954.patch

Aside from that, I heard a lot of discussion about Rules being the way to go, so I tried Workflow rules, but I got some errors when trying to create the rules, it kept asking for a node but I wanted to chose the content type and the condition to be the workflow state change but it kept giving me a white screen with an error message so I then disabled workflow rules and tried workflow action and trigger rather than spend time debugging the workflow rules, not sure yet how I will get the workflows to work or if I'll figure out another way to trigger an e-mail message notification when the state of the workflow changes from draft to comms, I'll have to do some more testing.

TODO:

joejoseph00 commented 12 years ago

Added a couple views today, PD Contract and PD Reclassification. Looking pretty close to the D6 views. Admin views next week, those will be much more challenging. More workflow tests soon.

joejoseph00 commented 12 years ago

add to dependancy: enable php filter module (core)

joejoseph00 commented 12 years ago

I am now looking into the public facing section of this. In D6 we were using the facetted search module. This does not exist in D7, solr which is a great product is overkill for PD in my opinion , and to simplify the deployment, I'd like to find a simpler solution, as I doubt that any department has millions of contracts. Our small agency has only hundreds at max since a couple years. So performance is not really an issue for the scope of PD . Looking into this shortly. Apparently the http://drupal.org/project/search_api module might be the way to go, can use Solr, or native db.

finweb commented 12 years ago

Have you try this module for Drupal 7. http://drupal.org/project/facetapi It worked with search_api and search_api_db.

joejoseph00 commented 12 years ago

Hi @finweb , thanks for the heads up, I just installed Search API, and it looks like the facet api is a requirement for Search facets. I'll give this a shot and see what happens.

sylus commented 12 years ago

Hey facetapi is an awesome module and is part of the distro. As for search_api this will require further discussion as I believe apachesolr is further along and better supported for use with solr itself. Search_API focuses on many search platforms.

Although we are based on panopoly and I could be persuaded to bring in: http://drupal.org/project/panopoly_search which uses search_api and search_api_db

joel-osc commented 12 years ago

An interesting approach here: http://envisioninteractive.com/drupal/drupal-7-views-with-faceted-filters-without-apachesolr/

joejoseph00 commented 12 years ago

for PD Contract we need to facet by year then month

Travel and Hospitality it's by member names and former members names and then facet to expense type and date

for reclassifications it's by date or "Organizational Component"

Note: we don't have a grants and contributions program, so we don't need to do PD for that and not sure if my boss will allow me to take time to do this as we don't need it.

joejoseph00 commented 12 years ago

Thanks everyone , wow, quick responses. Great suggestions. Will investigate

joejoseph00 commented 12 years ago

as I was trying to import contracts using feeds, ( so that I can have useful data to test the facetted search that I will be setting up,) when doing this, I found a bug in feeds where it doesn't see computed fields and it doesn't see money/currency fields. Currently investigating the Fields API to see if it's a feeds issue or a field issue specific to computed field and money/currency fields. Might have to patch feeds module, will definately be wanting to use the feeds module. Either that or patch computed field or money/currency field.

StephenOTT commented 12 years ago

There was some work done on the money/currency fields in 2011 for feeds integration. There is a patch that may be a start for you: http://drupal.org/node/628354

joejoseph00 commented 12 years ago

Thanks @StephenOTT , although that patch is for D6, I'll see if I can work it into D7 or else find another D7 patch. Currently I'm looking at the feeds api to see if there's anything obvious. I did notice that the table columns that work for other fields data columns end in _value , but in money/currency it ends in _amount . I tried a quick hack to create that row in the database and populate it, but that didn't work so I've got a simple custom field to test that is fully documented in the book "pro drupal development" (for D7) and will test it against feeds and see if/why it works and why money/currency doesn't. Or else find a D7 patch. I'm using some knockoff money/currency field. Also, computed field doesn't work either, so if I figure out why it doesn't work I can fix computed field as well.

These are the field modules I'm having trouble with: D7 money - http://drupal.org/sandbox/nouriassafi/1619328 Money field

computed field - http://ftp.drupal.org/files/projects/computed_field-7.x-1.0-beta1.tar.gz

joejoseph00 commented 12 years ago

having same feeds problem with this example field, so either it's missing a hook, or there's maybe a feeds module problem, so I'm going to look for the missing hook tomorrow. Basically, feeds does not see this when trying to set a mapping to it. Feeds only seems to see the core fields that I have, but isn't seeing the custom fields according to my preliminary testing. Here's the example field: http://www.apress.com/downloadable/download/sample/sample_id/261/ chapter 8 page 171 of pro drupal development third edition "working with fields" color example field this D7 example used to be on Drupal.org here:http://api.drupal.org/api/drupal/developer--examples--color--color_example--color_example.module/7 , but is now only available here: http://www.apress.com/downloadable/download/sample/sample_id/261/

as per http://www.drupalbook.com/errata3?page=1

eleclerc commented 12 years ago

the patch contain the answer. The money module (or any custom field for that matter) needs to implement a hook to let the feed module know about this field. In D7 case, it's hook_feeds_processor_targets_alter() that needs to be implemented

joejoseph00 commented 12 years ago

Alors il va falloir patcher plusieurs modules (money, computed field, et...color example)

eleclerc commented 12 years ago

it depends, patch could be sent upstream to be integrated, or the hooks could be defined in your mysite.module (that would cause issue though if, let's say for example, money add the feeds integration upstream)

joejoseph00 commented 12 years ago

I got the feeds integration to work by adding these functions to the D7 money field module: http://wet.7pro.ca/sites/wet.7pro.ca/files/money.diff , mostly copy pasted from the commerce feeds module, and from the feeds API documentation. From my early tests it works, but will have to test it more thoroughly. I'll try to get this code going upstream, the developer of this module responded friday so he should get this. Would save us from having to put the code in the pd module.

joejoseph00 commented 11 years ago

UPDATE: By following the envisioninteractive.com tutorial I did manage to get a primitive Facetted search using the built in database search engine and using a couple search api and facet api modules. It's pretty cool, able to make facetted searches on most columns very easily but having an issue with the date column (very important to PD) . For some reason the date column only works in integer mode, otherwise an error occurs. I might have to do some minor customized code or patching here and there around the edges unless I figure out an easier way. Also, for some reason the money module I'm using is not showing up in the indexed field listings so I can't add it to the view. Again, as I solved this problem with the feeds module, I imagine some custom code will have to be written for that too. Probably very minor , if it's much trouble I'll probably switch to decimal and worry about dollar sign formatting and locale currency formatting using other "ways". So, I've updated the PD feature, latest version is alpha 1.9 (link to the file is in the previous posts), it's missing only the money.diff and if anyone wants to try it they might want to give me a shout for a quick rundown.

joejoseph00 commented 11 years ago

Found the solution to the bug with Database search causing an issue in facet api and date field in my environment, there's a patch.

http://drupal.org/node/1550772

http://drupal.org/files/1550772--facets-no-limit-1.patch Note: this patch should make it into newer versions of search api db

Project: Search API » Database search Version: 7.x-1.0 » 7.x-1.x-dev Component: Facets » Code

Using the above patch I got the date field facet to work correctly, it facets by year, then drills down to month, then day, then hour

I also put a facet on vendor, so you can drill down (facet) contracts by vendor

I will work on this more, then create an updated feature with all these new facets.

TODO: for some reason facet_api is not seeing money fields. Will probably have to patch that myself.

StephenOTT commented 11 years ago

@joejoseph00 Glad to see you making progress!

joejoseph00 commented 11 years ago

sorry for delay, still working on peripheral items, doing testing

I just tested the feature in the latest WetBoew build from yesterday and I get this error after viewing one of the 1400 contract nodes I imported:

I go to content , click on a node (content) and this message comes up: Not sure what it means:

Error message

joejoseph00 commented 11 years ago

Whether it's in english or french, any node , anything I imported , they all give this message. I didn't get this message on previous builds.

sylus commented 11 years ago

Yeah this is entity translation ignore it for now as et isn't configured for nodes. Minor warning I will fix soon

Sent from my iPhone

On 2012-09-07, at 3:13 PM, J Olstad notifications@github.com wrote:

Whether it's in english or french, any node , anything I imported , they all give this message

— Reply to this email directly or view it on GitHub.

joejoseph00 commented 11 years ago

Thank you @sylus, to his smart phone ! :)

joejoseph00 commented 11 years ago

Hi @sylus, I've tested the database search api module, the dev version that is included in this release, it has a problem with my date field (CONTRACT_DATE).

Previously I tested the same index and feature to be working with the prod release of search api db: http://ftp.drupal.org/files/projects/search_api_db-7.x-1.0-beta3.tar.gz and the patch: http://drupal.org/files/1550772--facets-no-limit-1.patch

but this dev version and the patches we are running are not working.

here's where it fails: line 334 of service.inc $boost = $fields[$name]['boost'];

I haven't ran any traces on this yet.

wetboew/profiles/wetkit/modules/contrib/search_api_db/service.inc

Here's the error I get when I try to index Notice: Undefined index: field_contract_date in SearchApiDbService->indexItem() (line 334 of /home/JOlstad/public_html/wetboew/profiles/wetkit/modules/contrib/search_api_db/service.inc). Couldn't index items. Check the logs for details.

here's the watchdog

Type: search_api_db Date: Friday, September 7, 2012 - 4:26pm User: admin Location: http://vl-webdev-01/~JOlstad/wetboew/admin/config/search/search_api/index/index_pd_contract/status Referrer: http://vl-webdev-01/~JOlstad/wetboew/admin/config/search/search_api/index/index_pd_contract/status Message: SQLSTATE[42000]: Syntax error or access violation: 1064 You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'WHERE (item_id = '192153')' at line 1 Severity: warning Hostname: 10.10.20.67

joejoseph00 commented 11 years ago

Hi @Sylus, I did some more testing, a date index done by search api db fails on line 334 of service.inc when "wetkit search" is enabled. However, if I disable "wetkit search", the index will create properly and the facetted search by date will work. If "wetkit search" is enabled, then it fails to index the date with the error noted in my previous message. If you want, I could set up a demo to show you what I'm talking about.

Is there a hook in "Wetkit Search" that is somehow altering the behaviour of "search db api" ? Based on this empirical evidence, it appears that there something in wetkit search that is affecting search db api but I don't know exactly what. I'm not sure what "Wetkit Search" does.

Thanks,

joejoseph00 commented 11 years ago

Hi @Sylus, should we create a new issue for "wetkit search"?

sylus commented 11 years ago

Hmm not sure let me look into this. WetKit Search does a fair amount of stuff. It likely won't be a hard fix I just need to know what is going on a bit more and have the test environment using your "app"

joejoseph00 commented 11 years ago

What is "Wetkit Search"? is it just a feature? or is it a module? I'm in the process of setting this up on an external site so that you can maybe see what I'm talking about and try it for yourself.

sylus commented 11 years ago

WetKit Search controls the whole Search Interface and makes it panelized. It also integrates with apache solr if detected or uses search api db. It is both a module and a feature. Arguably one of the cooler ones we have.

joejoseph00 commented 11 years ago

I'll have a demo ready for you in a couple hours.

joejoseph00 commented 11 years ago

@sylus, I sent you an e-mail with the admin access info. Here's a link for everyone else http://198.103.146.168/~JOlstad/wetboew/indextest

sylus commented 11 years ago

Hey @joejoseph00 can you give me a snapshot of the call stack? I can probably look at this later in the week otherwise.

joejoseph00 commented 11 years ago

Hi @sylus, good news (knock on wood), I was trying to get a snapshot of the call stack, but when I re-enabled the Wetkit Search module, I could not repeat the issue. The index re-indexed properly with no errors and the facetted search continued to work.

I will try another test, doing a fresh install from scratch and installing my feature and various dependancies, which is how I got the error in the first place.

if it comes up again, I'll let you know.

BTW, for doing a stack trace, do you use the http://drupal.org/project/drupalforfirebug drupal for firebug? Or do you use more quick and dirty methods like modifying the bootstrap.inc as documented here: http://www.lullabot.com/articles/quick-and-dirty-debugging , or some other method /tool ?

Thanks again,

joejoseph00 commented 11 years ago

Hi @sylus, I've done more changes to the pd feature and had to test it in a new build, so I grabbed a debug backtrace when I came accross this issue again.

If you're curious, I did capture a debug_backtrace , not sure if this is what you call a "snapshot of the call stack".

I did this by modifying the bootstrap.inc as documented http://www.lullabot.com/articles/quick-and-dirty-debugging

here is the 700 kb backtrace (below) http://wet.7pro.ca/sites/wet.7pro.ca/files/stack_trace.txt

Aside from that, I am currently working on changing some field_types in the feature, I hope to be able to get this done today to show some progress.

joejoseph00 commented 11 years ago

@sylus , ok, the workaround is working but I forgot an important step. If you're indexing a date field, change it to integer from the fields tab of view index. Index it, then go back into the view index "field" tab (of "search api") and change it from integer back to date.

I'll do some more investigating. Might not need to disable wetkit search at all, but it is probably a bug in search api , more investigation is required. For now, it's just an annoyance/inconvenience.

joejoseph00 commented 11 years ago

Have removed money, currency api, formatted number from the next version of this feature (version 13) . Replaced with a custom formatter for Canadian Dollars , this will allow for greater compatibility between contributed modules, rather than using a contributed module that adds a cck field we take a core field such as decimal and format it with a formatter using the "Custom Formatters version 7.x-2.2" module. The money module is no longer needed in our case as we just need to display one currency type but in two different localisations $9.00 (english) and 9,00$ (français).

This greatly simplifies implementation while improving localisation.

joejoseph00 commented 11 years ago

Will post the new version of this feature shortly.

joejoseph00 commented 11 years ago

Doing some cleanup, trying to panopolize this feature, implementing a few panopoly best practices that is giving me some extra work. The feature has been simplified, still having a few issues with the Display Suite to reproduce some date range fields, will work on this tomorrow, hope to have an update by friday sept 28 posted back here. Also going to look into the feeds issue where it lacks a gui interface for mapping the tnid. Will have to figure out a workaround for that , more details later.

joejoseph00 commented 11 years ago

according to Panopoly, they recommend disabling the pager in views by default, and setting up record limits instead, but do not really provide much of an explanation as to why or in what context. See the "use pager" section below:

http://drupal.org/node/1704274

joel-osc commented 11 years ago

I think this is is a best practice not to have pagers inside of panes, best to have a more link to a page view of the rest of the items and a lot more filters or exposed facets. But, that is just a general design recommendation, there are specific cases in which pagers would be nice within the pane which is why in the Allowed settings they recommend checking pager, so that through panels you can turn paging on. One thing to consider when doing this it is that it is likely best to enable ajax on the pager in views so that it works nicely within the pane.